Jul 31 12:15:11.903559 ip-10-0-136-35 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 12:15:12.277390 ip-10-0-136-35 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 12:15:12.277390 ip-10-0-136-35 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 12:15:12.277390 ip-10-0-136-35 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 12:15:12.277390 ip-10-0-136-35 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 12:15:12.277390 ip-10-0-136-35 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 12:15:12.279230 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.279144    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 12:15:12.282656 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282636    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 12:15:12.282656 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282654    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 12:15:12.282656 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282662    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282667    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282671    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282676    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282680    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282685    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282689    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282693    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282697    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282702    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282705    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282709    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282713    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282717    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282721    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282725    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282729    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282733    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282737    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282741    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 12:15:12.282858 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282745    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282749    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282753    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282758    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282761    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282766    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282770    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282775    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282780    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282785    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282789    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282793    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282797    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282801    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282814    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282818    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282824    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282828    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282833    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282837    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 12:15:12.283701 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282841    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282845    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282850    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282854    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282858    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282862    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282866    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282870    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282874    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282878    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282882    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282886    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282890    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282895    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282899    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282904    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282908    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282912    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282916    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 12:15:12.284567 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282920    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282924    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282928    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282932    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282936    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282940    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282944    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282950    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282954    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282959    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282963    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282968    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282972    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282976    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282981    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282985    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282989    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282993    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.282997    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283004    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 12:15:12.285376 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283008    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283012    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283016    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283021    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283025    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283029    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283033    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283038    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283041    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283048    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283055    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283061    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283065    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283070    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283074    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283078    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283083    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.283087    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284342    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 12:15:12.286276 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284353    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284359    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284364    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284369    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284374    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284378    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284383    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284387    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284391    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284395    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284399    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284403    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284407    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284411    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284416    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284420    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284424    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284428    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284433    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284437    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 12:15:12.287095 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284441    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284445    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284449    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284454    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284458    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284477    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284481    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284485    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284489    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284493    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284497    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284502    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284510    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284517    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284521    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284525    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284529    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284535    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284540    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284545    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 12:15:12.287879 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284549    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284553    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284558    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284562    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284566    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284570    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284574    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284578    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284581    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284585    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284589    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284593    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284600    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284606    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284611    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284615    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284620    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284625    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284632    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 12:15:12.288535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284638    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284644    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284650    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284654    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284659    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284665    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284669    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284674    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284677    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284681    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284686    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284690    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284696    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284700    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284704    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284709    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284713    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284719    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284722    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284726    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 12:15:12.289087 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284730    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284735    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284739    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284743    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284747    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284752    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284756    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284760    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284764    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284768    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284773    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284777    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284781    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284785    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284790    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284794    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284799    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284804    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.284809    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285407    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 12:15:12.289598 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285416    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285421    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285425    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285430    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285435    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285439    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285443    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285447    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285452    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285457    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285461    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285465    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285470    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285474    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285478    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285482    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285485    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285490    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285494    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285498    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 12:15:12.290185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285503    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285533    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285539    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285544    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285549    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285554    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285558    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285562    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285568    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285574    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285579    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285584    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285590    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285594    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285599    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285603    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285612    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285617    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285621    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285625    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 12:15:12.290704 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285630    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285634    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285638    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285643    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285647    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285651    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285655    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285660    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285664    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285668    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285672    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285676    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285681    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285685    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285690    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285694    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285698    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285702    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285708    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285712    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 12:15:12.291207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285716    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285721    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285727    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285733    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285738    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285742    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285747    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285751    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285756    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285761    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285765    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285770    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285774    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285778    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285782    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285787    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285791    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285796    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285801    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285805    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 12:15:12.291740 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285809    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285813    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285817    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285823    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285827    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285834    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285840    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285846    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285852    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285856    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285861    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285865    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285870    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285874    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285878    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285883    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285888    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.285893    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.285987    2586 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.285998    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 12:15:12.292366 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286010    2586 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286016    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286024    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286030    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286037    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286043    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286048    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286053    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286059    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286065    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286070    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286075    2586 flags.go:64] FLAG: --cgroup-root=""
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286079    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286084    2586 flags.go:64] FLAG: --client-ca-file=""
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286089    2586 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286094    2586 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286101    2586 flags.go:64] FLAG: --cluster-domain=""
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286109    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286114    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286120    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286125    2586 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286131    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286135    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 12:15:12.293038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286140    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286146    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286151    2586 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286156    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286160    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286165    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286171    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286177    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286182    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286187    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286192    2586 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286197    2586 flags.go:64] FLAG: --enable-server="true"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286203    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286211    2586 flags.go:64] FLAG: --event-burst="100"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286216    2586 flags.go:64] FLAG: --event-qps="50"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286221    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286226    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286232    2586 flags.go:64] FLAG: --eviction-hard=""
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286237    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286242    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286247    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286251    2586 flags.go:64] FLAG: --eviction-soft=""
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286257    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286277    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286282    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 12:15:12.293984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286287    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286293    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286298    2586 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286302    2586 flags.go:64] FLAG: --feature-gates=""
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286308    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286313    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286318    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286324    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286329    2586 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286335    2586 flags.go:64] FLAG: --help="false"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286340    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286345    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286349    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286354    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286360    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286366    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286371    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286375    2586 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286380    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286385    2586 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286391    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286397    2586 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286402    2586 flags.go:64] FLAG: --kube-reserved=""
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286407    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 12:15:12.294905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286412    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286417    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286422    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286427    2586 flags.go:64] FLAG: --lock-file=""
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286431    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286436    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286442    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286451    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286456    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286463    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286467    2586 flags.go:64] FLAG: --logging-format="text"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286472    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286477    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286482    2586 flags.go:64] FLAG: --manifest-url=""
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286487    2586 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286494    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286499    2586 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286506    2586 flags.go:64] FLAG: --max-pods="110"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286511    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286516    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286520    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286526    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286531    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286537    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286541    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 12:15:12.295531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286552    2586 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286557    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286562    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286567    2586 flags.go:64] FLAG: --pod-cidr=""
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286574    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286583    2586 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286588    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286593    2586 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286598    2586 flags.go:64] FLAG: --port="10250"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286603    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286608    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e3a684b7d25bfddc"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286613    2586 flags.go:64] FLAG: --qos-reserved=""
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286618    2586 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286623    2586 flags.go:64] FLAG: --register-node="true"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286628    2586 flags.go:64] FLAG: --register-with-taints=""
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286634    2586 flags.go:64] FLAG: --registry-burst="10"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286640    2586 flags.go:64] FLAG: --registry-qps="5"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286645    2586 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286650    2586 flags.go:64] FLAG: --reserved-memory=""
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286656    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286661    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286666    2586 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286671    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286675    2586 flags.go:64] FLAG: --runonce="false"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286680    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 12:15:12.296180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286685    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286690    2586 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286695    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286700    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286706    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286711    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286716    2586 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286720    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286725    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286730    2586 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286735    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286741    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286748    2586 flags.go:64] FLAG: --system-cgroups=""
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286753    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286762    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286766    2586 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286771    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286778    2586 flags.go:64] FLAG: --tls-min-version=""
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286782    2586 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286788    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286792    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286797    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286802    2586 flags.go:64] FLAG: --v="2"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286818    2586 flags.go:64] FLAG: --version="false"
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286824    2586 flags.go:64] FLAG: --vmodule=""
Jul 31 12:15:12.296807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286830    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.286835    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287016    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287023    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287027    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287032    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287037    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287041    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287045    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287050    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287056    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287062    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287068    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287073    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287077    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287082    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287087    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287091    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287096    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 12:15:12.297435 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287104    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287111    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287116    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287121    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287126    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287131    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287136    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287141    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287145    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287150    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287154    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287160    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287164    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287169    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287173    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287178    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287182    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287186    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287190    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287194    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 12:15:12.297902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287199    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287203    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287207    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287211    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287215    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287220    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287224    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287228    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287232    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287236    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287240    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287245    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287250    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287255    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287285    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287290    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287294    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287299    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287303    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287309    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 12:15:12.298420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287315    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287319    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287324    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287330    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287334    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287338    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287342    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287346    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287351    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287355    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287359    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287363    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287367    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287371    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287375    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287380    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287384    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287389    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287393    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287397    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 12:15:12.298928 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287401    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287406    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287410    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287414    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287420    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287425    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287429    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287441    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287445    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287449    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287454    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287458    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287462    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287466    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287471    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287476    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287481    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287485    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287489    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 12:15:12.299503 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287493    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287497    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.287502    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.287511    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.295897    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.295913    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.296738    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.296749    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296808    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296813    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296817    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296821    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296824    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296828    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296830    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 12:15:12.300005 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296833    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296835    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296838    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296841    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296844    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296846    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296849    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296852    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296864    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296867    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296869    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296872    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296875    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296877    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296880    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296883    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296886    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296888    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296891    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296893    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 12:15:12.300410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296896    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296899    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296902    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296905    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296908    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296910    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296913    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296916    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296919    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296921    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296924    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296927    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296930    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296932    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296935    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296939    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296942    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296945    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296948    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296952    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 12:15:12.300904 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296955    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296958    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296961    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296964    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296967    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296969    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296972    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296975    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296978    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296980    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296983    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296986    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296989    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296992    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296996    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.296999    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297002    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297005    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297008    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297011    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 12:15:12.301480 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297014    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297017    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297020    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297022    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297025    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297029    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297033    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297035    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297038    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297041    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297043    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297045    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297049    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297051    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297054    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297056    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297059    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297061    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297064    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 12:15:12.302079 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297066    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297069    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297073    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297076    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297080    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297083    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297086    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297089    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297091    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297094    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297097    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297099    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297102    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.297107    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 12:15:12.302604 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297220    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297225    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297228    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297231    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297233    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297236    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297239    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297241    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297244    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297247    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297249    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297252    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297255    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297258    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297261    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297276    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297279    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297282    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297285    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297287    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297290    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 12:15:12.303189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297292    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297295    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297298    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297300    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297303    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297306    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297309    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297311    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297314    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297317    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297319    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297322    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297325    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297328    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297331    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297333    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297336    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297338    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297341    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297343    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 12:15:12.303902 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297346    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297348    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297351    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297353    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297356    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297359    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297362    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297364    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297367    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297369    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297372    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297376    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297379    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297382    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297385    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297388    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297391    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297393    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297396    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 12:15:12.304504 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297398    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297401    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297404    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297406    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297409    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297411    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297414    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297416    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297419    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297421    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297424    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297427    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297429    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297432    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297434    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297437    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297440    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297442    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297445    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297448    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 12:15:12.305038 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297451    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297453    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297456    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297458    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297461    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297464    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297466    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297469    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297471    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297474    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297477    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297480    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297484    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297487    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297490    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297493    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297497    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297500    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 12:15:12.305678 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:12.297503    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 12:15:12.306117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.297508    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 12:15:12.306117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.297627    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 12:15:12.306117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.299575    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 12:15:12.306117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.300825    2586 server.go:1019] "Starting client certificate rotation"
Jul 31 12:15:12.306117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.300935    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 12:15:12.306117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.300976    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 12:15:12.306117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.303529    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 12:15:12.306117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.304647    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 12:15:12.312093 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.312075    2586 log.go:25] "Validated CRI v1 runtime API"
Jul 31 12:15:12.314209 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.314193    2586 log.go:25] "Validated CRI v1 image API"
Jul 31 12:15:12.315031 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.315018    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 12:15:12.318161 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.318142    2586 fs.go:135] Filesystem UUIDs: map[4a52fe74-d4bf-4569-9ae9-e6577f80bdbb:/dev/nvme0n1p4 61013e96-7a2b-4dc6-92c4-66c128de93b8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 31 12:15:12.318217 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.318160    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 12:15:12.323530 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.323420    2586 manager.go:216] Machine: {Timestamp:2026-07-31 12:15:12.32229266 +0000 UTC m=+0.309514191 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122956 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2735f52b257fa6afb9d0dc7b71fdad SystemUUID:ec2735f5-2b25-7fa6-afb9-d0dc7b71fdad BootID:88f50e9e-d612-4efa-b6fb-89c6eefcd34d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c9:85:0a:7f:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c9:85:0a:7f:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:43:c1:31:12:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 12:15:12.323530 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.323527    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 12:15:12.323641 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.323599    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 12:15:12.326708 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.326685    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 12:15:12.326886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.326719    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-35.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 12:15:12.327483 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.327471    2586 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 12:15:12.327516 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.327487    2586 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 12:15:12.327516 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.327494    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 12:15:12.328121 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.328110    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 12:15:12.328920 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.328908    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 12:15:12.330385 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.330370    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 12:15:12.332052 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.332040    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 12:15:12.332479 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.332464    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 12:15:12.332540 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.332504    2586 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 12:15:12.332583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.332542    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 12:15:12.332583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.332570    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 12:15:12.332583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.332583    2586 kubelet.go:398] "Adding apiserver pod source"
Jul 31 12:15:12.332698 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.332595    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 12:15:12.333521 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.333218    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 12:15:12.333521 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.333243    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 12:15:12.336379 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.336360    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 12:15:12.337729 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.337716    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 12:15:12.337773 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.337745    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 12:15:12.338776 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338764    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 12:15:12.338829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338780    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 12:15:12.338829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338786    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 12:15:12.338829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338821    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 12:15:12.338929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338835    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 12:15:12.338929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338842    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 12:15:12.338929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338850    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 12:15:12.338929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338856    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 12:15:12.338929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338863    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 12:15:12.338929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338868    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 12:15:12.338929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338876    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 12:15:12.338929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.338882    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 12:15:12.339551 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.339541    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 12:15:12.339551 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.339551    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 12:15:12.342869 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.342853    2586 server.go:1268] "Started kubelet"
Jul 31 12:15:12.342991 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.342935    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 12:15:12.343057 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.343014    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 12:15:12.343057 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.342943    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 12:15:12.343677 ip-10-0-136-35 systemd[1]: Started Kubernetes Kubelet.
Jul 31 12:15:12.344100 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.344083    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 12:15:12.346146 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.346131    2586 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 12:15:12.346822 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.346792    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-35.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 12:15:12.346982 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.346960    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 12:15:12.347035 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.346966    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 12:15:12.348133 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.347025    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-35.ec2.internal.18c75ee4669d3446  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-35.ec2.internal,UID:ip-10-0-136-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-35.ec2.internal,},FirstTimestamp:2026-07-31 12:15:12.342824006 +0000 UTC m=+0.330045538,LastTimestamp:2026-07-31 12:15:12.342824006 +0000 UTC m=+0.330045538,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-35.ec2.internal,}"
Jul 31 12:15:12.348552 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.348535    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 12:15:12.349318 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.349305    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 12:15:12.349387 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.349326    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 12:15:12.349387 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.349373    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:12.350042 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350026    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 12:15:12.350162 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350144    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 12:15:12.350162 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350030    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 12:15:12.350162 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350090    2586 factory.go:55] Registering systemd factory
Jul 31 12:15:12.350387 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350180    2586 factory.go:223] Registration of the systemd container factory successfully
Jul 31 12:15:12.350387 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350300    2586 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 12:15:12.350387 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350310    2586 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 12:15:12.350531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350439    2586 factory.go:153] Registering CRI-O factory
Jul 31 12:15:12.350531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350450    2586 factory.go:223] Registration of the crio container factory successfully
Jul 31 12:15:12.350531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350508    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 12:15:12.350648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350542    2586 factory.go:103] Registering Raw factory
Jul 31 12:15:12.350648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.350558    2586 manager.go:1196] Started watching for new ooms in manager
Jul 31 12:15:12.351186 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.351169    2586 manager.go:349] Starting recovery of all containers
Jul 31 12:15:12.360560 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.360528    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 12:15:12.360656 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.360513    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 12:15:12.362507 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.362401    2586 manager.go:354] Recovery completed
Jul 31 12:15:12.366676 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.366659    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 12:15:12.368727 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.368709    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 12:15:12.368800 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.368744    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 12:15:12.368800 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.368758    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID"
Jul 31 12:15:12.369256 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.369230    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 12:15:12.369256 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.369246    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 12:15:12.369256 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.369232    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 12:15:12.369444 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.369275    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 12:15:12.369926 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.369911    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 12:15:12.370002 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.369937    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 12:15:12.370002 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.369952    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID"
Jul 31 12:15:12.370501 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.370489    2586 policy_none.go:49] "None policy: Start"
Jul 31 12:15:12.370547 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.370505    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 12:15:12.370547 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.370515    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 12:15:12.371654 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.371642    2586 policy_none.go:47] "Start"
Jul 31 12:15:12.372306 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.372290    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h9zch"
Jul 31 12:15:12.379623 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.379606    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h9zch"
Jul 31 12:15:12.380210 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.380146    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-35.ec2.internal.18c75ee4682872b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-35.ec2.internal,UID:ip-10-0-136-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-35.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-35.ec2.internal,},FirstTimestamp:2026-07-31 12:15:12.368726705 +0000 UTC m=+0.355948241,LastTimestamp:2026-07-31 12:15:12.368726705 +0000 UTC m=+0.355948241,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-35.ec2.internal,}"
Jul 31 12:15:12.410580 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.410567    2586 manager.go:337] "Starting Device Plugin manager"
Jul 31 12:15:12.410647 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.410597    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 12:15:12.410647 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.410606    2586 server.go:87] "Starting device plugin registration server"
Jul 31 12:15:12.410881 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.410865    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 12:15:12.410950 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.410884    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 12:15:12.411065 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.410997    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 12:15:12.411111 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.411098    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 12:15:12.411111 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.411109    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 12:15:12.411627 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.411603    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 12:15:12.411723 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.411661    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:12.477023 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.476989    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 12:15:12.478119 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.478093    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 12:15:12.478221 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.478134    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 12:15:12.478221 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.478161    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 12:15:12.478221 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.478208    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 12:15:12.480210 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.480194    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 12:15:12.511068 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.511037    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 12:15:12.512174 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.512161    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 12:15:12.512231 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.512200    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 12:15:12.512231 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.512213    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID"
Jul 31 12:15:12.512315 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.512234    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.520673 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.520659    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.520725 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.520679    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-35.ec2.internal\": node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:12.544870 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.544816    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:12.578484 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.578463    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal"]
Jul 31 12:15:12.578535 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.578524    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 12:15:12.579886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.579871    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 12:15:12.579942 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.579898    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 12:15:12.579942 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.579909    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID"
Jul 31 12:15:12.581029 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.581018    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 12:15:12.581124 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.581113    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.581155 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.581138    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 12:15:12.581652 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.581635    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 12:15:12.581702 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.581673    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 12:15:12.581702 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.581675    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 12:15:12.581702 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.581682    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID"
Jul 31 12:15:12.581702 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.581692    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 12:15:12.581702 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.581702    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID"
Jul 31 12:15:12.582720 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.582707    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.582778 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.582731    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 12:15:12.583700 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.583687    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 12:15:12.583747 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.583715    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 12:15:12.583747 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.583726    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID"
Jul 31 12:15:12.615868 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.615851    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-35.ec2.internal\" not found" node="ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.619982 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.619970    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-35.ec2.internal\" not found" node="ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.645316 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.645295    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:12.746169 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.746137    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:12.751383 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.751364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ee029fa264a2a647e1d318de5a3789c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"8ee029fa264a2a647e1d318de5a3789c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.751465 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.751396    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d64eb269ad67e5f98e8a744fc0f2cae5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" (UID: \"d64eb269ad67e5f98e8a744fc0f2cae5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.751465 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.751421    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ee029fa264a2a647e1d318de5a3789c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"8ee029fa264a2a647e1d318de5a3789c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.846726 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.846650    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:12.851918 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.851898    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ee029fa264a2a647e1d318de5a3789c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"8ee029fa264a2a647e1d318de5a3789c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.852007 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.851930    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ee029fa264a2a647e1d318de5a3789c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"8ee029fa264a2a647e1d318de5a3789c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.852007 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.851954    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d64eb269ad67e5f98e8a744fc0f2cae5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" (UID: \"d64eb269ad67e5f98e8a744fc0f2cae5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.852007 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.851995    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d64eb269ad67e5f98e8a744fc0f2cae5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" (UID: \"d64eb269ad67e5f98e8a744fc0f2cae5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.852007 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.851989    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ee029fa264a2a647e1d318de5a3789c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"8ee029fa264a2a647e1d318de5a3789c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.852132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.851996    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ee029fa264a2a647e1d318de5a3789c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"8ee029fa264a2a647e1d318de5a3789c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.918050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.918013    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.922822 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:12.922804    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:12.947564 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:12.947538    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:13.047958 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:13.047918    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:13.148403 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:13.148380    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:13.174665 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.174650    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 12:15:13.249233 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:13.249206    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:13.300610 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.300584    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 12:15:13.301146 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.300714    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 12:15:13.301146 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.300747    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 12:15:13.349610 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:13.349585    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:13.383074 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.383049    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 12:10:11 +0000 UTC" deadline="2028-03-01 14:18:46.371525844 +0000 UTC"
Jul 31 12:15:13.383074 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.383073    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13898h3m32.988455962s"
Jul 31 12:15:13.450372 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:13.450349    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:13.488648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.488631    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 12:15:13.489895 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.489872    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 12:15:13.489977 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.489917    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 12:15:13.489977 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.489931    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID"
Jul 31 12:15:13.491632 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:13.491604    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee029fa264a2a647e1d318de5a3789c.slice/crio-b5f590988289838633386dfb5a4dbd9a9734f41604778f061b5a8c054451f426 WatchSource:0}: Error finding container b5f590988289838633386dfb5a4dbd9a9734f41604778f061b5a8c054451f426: Status 404 returned error can't find the container with id b5f590988289838633386dfb5a4dbd9a9734f41604778f061b5a8c054451f426
Jul 31 12:15:13.491981 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:13.491966    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64eb269ad67e5f98e8a744fc0f2cae5.slice/crio-2b07c13a8744143806a0fb4c6468ca2656b00acfe530fd1e4f61123a8454f1f5 WatchSource:0}: Error finding container 2b07c13a8744143806a0fb4c6468ca2656b00acfe530fd1e4f61123a8454f1f5: Status 404 returned error can't find the container with id 2b07c13a8744143806a0fb4c6468ca2656b00acfe530fd1e4f61123a8454f1f5
Jul 31 12:15:13.497034 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.497021    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 12:15:13.550678 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:13.550645    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:13.650981 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:13.650951    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found"
Jul 31 12:15:13.699983 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.699932    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 12:15:13.747559 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.747542    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 12:15:13.750087 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.750074    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:13.761430 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.761414    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 12:15:13.762317 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.762304    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal"
Jul 31 12:15:13.769650 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:13.769634    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 12:15:14.334888 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.334861    2586 apiserver.go:52] "Watching apiserver"
Jul 31 12:15:14.341235 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.341213    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 12:15:14.341727 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.341594    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-plkjn","openshift-image-registry/node-ca-jl8fm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal","openshift-multus/network-metrics-daemon-cf94v","openshift-network-diagnostics/network-check-target-kh24k","openshift-network-operator/iptables-alerter-zdb4h","kube-system/global-pull-secret-syncer-pwtgz","kube-system/konnectivity-agent-jsrdj","kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal","openshift-multus/multus-additional-cni-plugins-k8sxf","openshift-multus/multus-lhqd7","openshift-ovn-kubernetes/ovnkube-node-2rv4d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6","openshift-cluster-node-tuning-operator/tuned-wh6xs"]
Jul 31 12:15:14.343156 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.343131    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.343248 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.343206    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:14.344274 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.344241    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.345362 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.345341    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.347061 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.347045    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:14.347157 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.347072    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 12:15:14.347157 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.347124    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:14.347460 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.347442    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 12:15:14.347523 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.347472    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 12:15:14.347788 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.347773    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fscsb\""
Jul 31 12:15:14.348837 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.348347    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 12:15:14.348837 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.348554    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgj52\""
Jul 31 12:15:14.348837 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.348754    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 12:15:14.351461 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.351444    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 12:15:14.353915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.353547    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:14.353915 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.353710    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:14.355668 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.355372    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.358623 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.358148    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:14.358623 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.358282    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkktr\""
Jul 31 12:15:14.358623 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.358609    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 12:15:14.358794 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.358643    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 12:15:14.358794 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.358778    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359698    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359729    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-socket-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359752    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-sys\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359772    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-run\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359790    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/827b42e8-32dd-4f96-93ee-fbbad611cc5f-tmp\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359817    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7-iptables-alerter-script\") pod \"iptables-alerter-zdb4h\" (UID: \"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7\") " pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359841    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-systemd\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359860    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-tuned\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359881    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359918    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwlr\" (UniqueName: \"kubernetes.io/projected/bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7-kube-api-access-gpwlr\") pod \"iptables-alerter-zdb4h\" (UID: \"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7\") " pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359939    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0a25e8a-583a-4194-a8df-ff565cac7a43-kubelet-config\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359959    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-device-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.359979    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-sysconfig\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360022    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgf4k\" (UniqueName: \"kubernetes.io/projected/827b42e8-32dd-4f96-93ee-fbbad611cc5f-kube-api-access-wgf4k\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360044    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:14.360534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360064    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-host\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360084    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7-host-slash\") pod \"iptables-alerter-zdb4h\" (UID: \"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7\") " pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360105    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-sys-fs\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360144    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360146    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-modprobe-d\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360190    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-sysctl-d\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360218    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360237    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-sysctl-conf\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360277    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-lib-modules\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360299    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzdg\" (UniqueName: \"kubernetes.io/projected/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-kube-api-access-dmzdg\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360320    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-registration-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360342    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0a25e8a-583a-4194-a8df-ff565cac7a43-dbus\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360380    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360403    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5rlb\" (UniqueName: \"kubernetes.io/projected/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-kube-api-access-j5rlb\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360426    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-kubernetes\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360448    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-var-lib-kubelet\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360581    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kmrbj\""
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360767    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.360777    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 12:15:14.361376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.361045    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.363348 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.362173    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 12:15:14.363348 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.362498    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tl586\""
Jul 31 12:15:14.363348 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.362510    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 12:15:14.363348 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.362695    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.363348 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.363249    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9zhn4\""
Jul 31 12:15:14.363562 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.363386    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 12:15:14.363562 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.363478    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 12:15:14.363562 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.363479    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 12:15:14.363742 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.363662    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.363742 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.363730    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 12:15:14.366285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.364838    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.366285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.365436    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 12:15:14.366285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.365509    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 12:15:14.366285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.365525    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 12:15:14.366285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.365564    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 12:15:14.366285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.365740    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 12:15:14.366285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.365769    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5bg96\""
Jul 31 12:15:14.366285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.365869    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nktfx\""
Jul 31 12:15:14.366285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.365923    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 12:15:14.367191 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.367173    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 12:15:14.367346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.367331    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 12:15:14.367692 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.367677    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 12:15:14.368138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.368120    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 12:15:14.368380 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.368360    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 12:15:14.368553 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.368535    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jn95k\""
Jul 31 12:15:14.368846 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.368794    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 12:15:14.381990 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.381973    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5l2jz"
Jul 31 12:15:14.390321 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.390300    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5l2jz"
Jul 31 12:15:14.451910 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.451888    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 12:15:14.460785 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.460762    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-daemon-config\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.460881 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.460791    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7-iptables-alerter-script\") pod \"iptables-alerter-zdb4h\" (UID: \"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7\") " pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.460881 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.460822    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-systemd\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.460881 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.460845    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-tuned\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.460881 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.460866    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:14.461029 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.460892    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-systemd\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.461029 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.460937    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jcl2\" (UniqueName: \"kubernetes.io/projected/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-kube-api-access-5jcl2\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.461029 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.460968    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1074883-7ab6-46a5-a22c-07e17d691672-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.461029 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.460990    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-system-cni-dir\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.461029 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461014    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-run-k8s-cni-cncf-io\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.461205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461035    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-cni-bin\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.461205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461060    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0a25e8a-583a-4194-a8df-ff565cac7a43-kubelet-config\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.461205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-sysconfig\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.461205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461105    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:14.461205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461127    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.461205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461156    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0a25e8a-583a-4194-a8df-ff565cac7a43-kubelet-config\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.461205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461158    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 12:15:14.461205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461195    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qlgq\" (UniqueName: \"kubernetes.io/projected/b1074883-7ab6-46a5-a22c-07e17d691672-kube-api-access-5qlgq\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.461205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461193    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-sysconfig\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.461230    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461235    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-cnibin\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461256    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-cni-binary-copy\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.461322    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:14.961285314 +0000 UTC m=+2.948506850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461353    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-env-overrides\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461378    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-modprobe-d\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461397    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-sysctl-d\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461410    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-slash\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461430    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-node-log\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461353    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7-iptables-alerter-script\") pod \"iptables-alerter-zdb4h\" (UID: \"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7\") " pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461482    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd73de9-7b12-4de5-ab9b-875c7bf9d62c-tmp-dir\") pod \"node-resolver-plkjn\" (UID: \"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c\") " pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461499    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-modprobe-d\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.461524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461527    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-var-lib-kubelet\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461538    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-sysctl-d\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-sysctl-conf\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461581    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-lib-modules\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461614    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-run-netns\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461630    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-sysctl-conf\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461642    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-run-openvswitch\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461654    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-lib-modules\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461665    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-ovn-node-metrics-cert\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461688    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhlq7\" (UniqueName: \"kubernetes.io/projected/70b1fc55-23f2-41da-981d-51d4f737c4e5-kube-api-access-jhlq7\") pod \"node-ca-jl8fm\" (UID: \"70b1fc55-23f2-41da-981d-51d4f737c4e5\") " pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461709    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-cnibin\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461732    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1074883-7ab6-46a5-a22c-07e17d691672-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461779    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-registration-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461809    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461833    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-ovnkube-script-lib\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461857    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1074883-7ab6-46a5-a22c-07e17d691672-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.462054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461877    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-registration-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461886    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-kubernetes\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.461908    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461909    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.461956    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret podName:e0a25e8a-583a-4194-a8df-ff565cac7a43 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:14.961943473 +0000 UTC m=+2.949164989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret") pod "global-pull-secret-syncer-pwtgz" (UID: "e0a25e8a-583a-4194-a8df-ff565cac7a43") : object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461983    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461986    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-socket-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.461995    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-kubernetes\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462009    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-sys\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462033    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bfd73de9-7b12-4de5-ab9b-875c7bf9d62c-hosts-file\") pod \"node-resolver-plkjn\" (UID: \"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c\") " pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462044    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-sys\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462055    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-run-netns\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462090    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-var-lib-cni-bin\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462100    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-socket-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462118    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rclgs\" (UniqueName: \"kubernetes.io/projected/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-kube-api-access-rclgs\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462151    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/827b42e8-32dd-4f96-93ee-fbbad611cc5f-tmp\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462177    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27c4f51d-6014-4c7c-8562-034f42cbf93e-agent-certs\") pod \"konnectivity-agent-jsrdj\" (UID: \"27c4f51d-6014-4c7c-8562-034f42cbf93e\") " pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:14.462768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462199    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-hostroot\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462228    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-socket-dir-parent\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462277    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwlr\" (UniqueName: \"kubernetes.io/projected/bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7-kube-api-access-gpwlr\") pod \"iptables-alerter-zdb4h\" (UID: \"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7\") " pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462301    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-device-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgf4k\" (UniqueName: \"kubernetes.io/projected/827b42e8-32dd-4f96-93ee-fbbad611cc5f-kube-api-access-wgf4k\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462342    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-log-socket\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462363    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-cni-dir\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462385    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-run-multus-certs\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462406    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-ovnkube-config\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462432    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-host\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462454    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7-host-slash\") pod \"iptables-alerter-zdb4h\" (UID: \"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7\") " pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462475    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-sys-fs\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462495    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-systemd-units\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462521    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-os-release\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462543    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462566    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzdg\" (UniqueName: \"kubernetes.io/projected/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-kube-api-access-dmzdg\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462588    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-etc-openvswitch\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.463515 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462611    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-run-ovn\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462634    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462654    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70b1fc55-23f2-41da-981d-51d4f737c4e5-host\") pod \"node-ca-jl8fm\" (UID: \"70b1fc55-23f2-41da-981d-51d4f737c4e5\") " pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462676    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-system-cni-dir\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462698    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462724    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0a25e8a-583a-4194-a8df-ff565cac7a43-dbus\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462759    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5rlb\" (UniqueName: \"kubernetes.io/projected/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-kube-api-access-j5rlb\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462774    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-device-dir\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462780    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-cni-netd\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462855    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-host\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462894    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7-host-slash\") pod \"iptables-alerter-zdb4h\" (UID: \"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7\") " pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.462942    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-sys-fs\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463032    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27c4f51d-6014-4c7c-8562-034f42cbf93e-konnectivity-ca\") pod \"konnectivity-agent-jsrdj\" (UID: \"27c4f51d-6014-4c7c-8562-034f42cbf93e\") " pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463250    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8jd\" (UniqueName: \"kubernetes.io/projected/bfd73de9-7b12-4de5-ab9b-875c7bf9d62c-kube-api-access-wv8jd\") pod \"node-resolver-plkjn\" (UID: \"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c\") " pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463292    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-var-lib-cni-multus\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463305    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0a25e8a-583a-4194-a8df-ff565cac7a43-dbus\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.464132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463326    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-etc-kubernetes\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463351    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-conf-dir\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463386    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-var-lib-kubelet\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463425    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-run-systemd\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463449    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-var-lib-openvswitch\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463494    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-var-lib-kubelet\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463506    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70b1fc55-23f2-41da-981d-51d4f737c4e5-serviceca\") pod \"node-ca-jl8fm\" (UID: \"70b1fc55-23f2-41da-981d-51d4f737c4e5\") " pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463535    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-run\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463573    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-kubelet\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463600    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-os-release\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.463682    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/827b42e8-32dd-4f96-93ee-fbbad611cc5f-run\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.464346    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/827b42e8-32dd-4f96-93ee-fbbad611cc5f-etc-tuned\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.465548 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.465527    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/827b42e8-32dd-4f96-93ee-fbbad611cc5f-tmp\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.466776 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.466758    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 12:15:14.466907 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.466782    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 12:15:14.466907 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.466796    2586 projected.go:196] Error preparing data for projected volume kube-api-access-n5z67 for pod openshift-network-diagnostics/network-check-target-kh24k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:14.466907 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.466846    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67 podName:e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c nodeName:}" failed. No retries permitted until 2026-07-31 12:15:14.96683159 +0000 UTC m=+2.954053109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n5z67" (UniqueName: "kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67") pod "network-check-target-kh24k" (UID: "e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:14.469562 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.469528    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgf4k\" (UniqueName: \"kubernetes.io/projected/827b42e8-32dd-4f96-93ee-fbbad611cc5f-kube-api-access-wgf4k\") pod \"tuned-wh6xs\" (UID: \"827b42e8-32dd-4f96-93ee-fbbad611cc5f\") " pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.469922 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.469852    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwlr\" (UniqueName: \"kubernetes.io/projected/bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7-kube-api-access-gpwlr\") pod \"iptables-alerter-zdb4h\" (UID: \"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7\") " pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.470620 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.470592    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzdg\" (UniqueName: \"kubernetes.io/projected/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-kube-api-access-dmzdg\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:14.470962 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.470944    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5rlb\" (UniqueName: \"kubernetes.io/projected/9009cfca-fe02-47d8-a473-75e9d6b0cd8e-kube-api-access-j5rlb\") pod \"aws-ebs-csi-driver-node-vt4v6\" (UID: \"9009cfca-fe02-47d8-a473-75e9d6b0cd8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.484067 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.484018    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" event={"ID":"8ee029fa264a2a647e1d318de5a3789c","Type":"ContainerStarted","Data":"b5f590988289838633386dfb5a4dbd9a9734f41604778f061b5a8c054451f426"}
Jul 31 12:15:14.485405 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.485381    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" event={"ID":"d64eb269ad67e5f98e8a744fc0f2cae5","Type":"ContainerStarted","Data":"2b07c13a8744143806a0fb4c6468ca2656b00acfe530fd1e4f61123a8454f1f5"}
Jul 31 12:15:14.563997 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.563967    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bfd73de9-7b12-4de5-ab9b-875c7bf9d62c-hosts-file\") pod \"node-resolver-plkjn\" (UID: \"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c\") " pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.564139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-run-netns\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564027    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-var-lib-cni-bin\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564050    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rclgs\" (UniqueName: \"kubernetes.io/projected/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-kube-api-access-rclgs\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564073    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27c4f51d-6014-4c7c-8562-034f42cbf93e-agent-certs\") pod \"konnectivity-agent-jsrdj\" (UID: \"27c4f51d-6014-4c7c-8562-034f42cbf93e\") " pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:14.564139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564081    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-run-netns\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564090    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bfd73de9-7b12-4de5-ab9b-875c7bf9d62c-hosts-file\") pod \"node-resolver-plkjn\" (UID: \"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c\") " pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.564139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564094    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-hostroot\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-socket-dir-parent\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564180    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-log-socket\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564194    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-cni-dir\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564209    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-run-multus-certs\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564226    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-hostroot\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564232    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-ovnkube-config\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564294    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-systemd-units\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564319    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-os-release\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564345    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-etc-openvswitch\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564355    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-systemd-units\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564366    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-run-ovn\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564382    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-log-socket\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564402    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564425    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-os-release\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564433    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-etc-openvswitch\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564447    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70b1fc55-23f2-41da-981d-51d4f737c4e5-host\") pod \"node-ca-jl8fm\" (UID: \"70b1fc55-23f2-41da-981d-51d4f737c4e5\") " pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564437    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-run-multus-certs\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.564494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564473    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-system-cni-dir\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564479    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-cni-dir\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564475    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564490    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-run-ovn\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564408    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-socket-dir-parent\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564506    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70b1fc55-23f2-41da-981d-51d4f737c4e5-host\") pod \"node-ca-jl8fm\" (UID: \"70b1fc55-23f2-41da-981d-51d4f737c4e5\") " pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564506    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564532    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-system-cni-dir\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564545    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-cni-netd\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564569    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27c4f51d-6014-4c7c-8562-034f42cbf93e-konnectivity-ca\") pod \"konnectivity-agent-jsrdj\" (UID: \"27c4f51d-6014-4c7c-8562-034f42cbf93e\") " pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564570    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-var-lib-cni-bin\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564598    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8jd\" (UniqueName: \"kubernetes.io/projected/bfd73de9-7b12-4de5-ab9b-875c7bf9d62c-kube-api-access-wv8jd\") pod \"node-resolver-plkjn\" (UID: \"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c\") " pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564624    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-var-lib-cni-multus\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564628    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564660    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-etc-kubernetes\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564684    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-var-lib-cni-multus\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564687    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-conf-dir\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564720    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-conf-dir\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564725    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-run-systemd\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564751    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-var-lib-openvswitch\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564754    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-etc-kubernetes\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564775    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70b1fc55-23f2-41da-981d-51d4f737c4e5-serviceca\") pod \"node-ca-jl8fm\" (UID: \"70b1fc55-23f2-41da-981d-51d4f737c4e5\") " pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564789    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-var-lib-openvswitch\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564798    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-kubelet\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564817    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-ovnkube-config\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564832    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-run-systemd\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564897    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-cni-netd\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564906    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-os-release\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564927    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-kubelet\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564930    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-daemon-config\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564955    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-os-release\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.564995    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcl2\" (UniqueName: \"kubernetes.io/projected/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-kube-api-access-5jcl2\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565017    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1074883-7ab6-46a5-a22c-07e17d691672-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565019    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27c4f51d-6014-4c7c-8562-034f42cbf93e-konnectivity-ca\") pod \"konnectivity-agent-jsrdj\" (UID: \"27c4f51d-6014-4c7c-8562-034f42cbf93e\") " pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565038    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-system-cni-dir\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.565829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565060    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-run-k8s-cni-cncf-io\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565098    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-cni-bin\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565132    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565158    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qlgq\" (UniqueName: \"kubernetes.io/projected/b1074883-7ab6-46a5-a22c-07e17d691672-kube-api-access-5qlgq\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565182    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-cnibin\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565204    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-cni-binary-copy\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565225    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-env-overrides\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565228    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70b1fc55-23f2-41da-981d-51d4f737c4e5-serviceca\") pod \"node-ca-jl8fm\" (UID: \"70b1fc55-23f2-41da-981d-51d4f737c4e5\") " pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565245    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-cni-bin\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565256    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-slash\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-node-log\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565308    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-cnibin\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565311    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-slash\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd73de9-7b12-4de5-ab9b-875c7bf9d62c-tmp-dir\") pod \"node-resolver-plkjn\" (UID: \"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c\") " pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565353    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565443    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-multus-daemon-config\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565504    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-system-cni-dir\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd73de9-7b12-4de5-ab9b-875c7bf9d62c-tmp-dir\") pod \"node-resolver-plkjn\" (UID: \"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c\") " pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.566492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565623    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-run-k8s-cni-cncf-io\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565656    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-node-log\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565684    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-var-lib-kubelet\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565710    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-run-netns\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565733    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-run-openvswitch\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565739    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-cni-binary-copy\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565757    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-ovn-node-metrics-cert\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565779    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhlq7\" (UniqueName: \"kubernetes.io/projected/70b1fc55-23f2-41da-981d-51d4f737c4e5-kube-api-access-jhlq7\") pod \"node-ca-jl8fm\" (UID: \"70b1fc55-23f2-41da-981d-51d4f737c4e5\") " pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565789    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-host-run-netns\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565801    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-cnibin\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565824    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1074883-7ab6-46a5-a22c-07e17d691672-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-ovnkube-script-lib\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565878    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-env-overrides\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565886    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1074883-7ab6-46a5-a22c-07e17d691672-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.565827    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-host-var-lib-kubelet\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.566466    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-run-openvswitch\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.566529    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1074883-7ab6-46a5-a22c-07e17d691672-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.567273 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.566784    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1074883-7ab6-46a5-a22c-07e17d691672-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.567952 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.566913    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27c4f51d-6014-4c7c-8562-034f42cbf93e-agent-certs\") pod \"konnectivity-agent-jsrdj\" (UID: \"27c4f51d-6014-4c7c-8562-034f42cbf93e\") " pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:14.567952 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.566960    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1074883-7ab6-46a5-a22c-07e17d691672-cnibin\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.567952 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.566960    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1074883-7ab6-46a5-a22c-07e17d691672-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.567952 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.567512    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-ovnkube-script-lib\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.571116 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.571087    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rclgs\" (UniqueName: \"kubernetes.io/projected/eb11a5be-943c-4a5c-8ce3-06b1fab1c840-kube-api-access-rclgs\") pod \"multus-lhqd7\" (UID: \"eb11a5be-943c-4a5c-8ce3-06b1fab1c840\") " pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.571382 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.571361    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-ovn-node-metrics-cert\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.571840 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.571819    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8jd\" (UniqueName: \"kubernetes.io/projected/bfd73de9-7b12-4de5-ab9b-875c7bf9d62c-kube-api-access-wv8jd\") pod \"node-resolver-plkjn\" (UID: \"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c\") " pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.572989 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.572973    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcl2\" (UniqueName: \"kubernetes.io/projected/c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223-kube-api-access-5jcl2\") pod \"ovnkube-node-2rv4d\" (UID: \"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.573200 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.573177    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qlgq\" (UniqueName: \"kubernetes.io/projected/b1074883-7ab6-46a5-a22c-07e17d691672-kube-api-access-5qlgq\") pod \"multus-additional-cni-plugins-k8sxf\" (UID: \"b1074883-7ab6-46a5-a22c-07e17d691672\") " pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.573542 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.573527    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhlq7\" (UniqueName: \"kubernetes.io/projected/70b1fc55-23f2-41da-981d-51d4f737c4e5-kube-api-access-jhlq7\") pod \"node-ca-jl8fm\" (UID: \"70b1fc55-23f2-41da-981d-51d4f737c4e5\") " pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.657609 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.657582    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6"
Jul 31 12:15:14.664471 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.664451    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wh6xs"
Jul 31 12:15:14.675778 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.675758    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdb4h"
Jul 31 12:15:14.682197 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.682178    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:14.690570 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.690550    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-plkjn"
Jul 31 12:15:14.703218 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.703201    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jl8fm"
Jul 31 12:15:14.711720 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.711703    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8sxf"
Jul 31 12:15:14.718137 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.718118    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lhqd7"
Jul 31 12:15:14.724612 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.724597    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:14.821930 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.821902    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 12:15:14.968645 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.968566    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:14.968645 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.968614    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:14.968848 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:14.968653    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:14.968848 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.968753    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 12:15:14.968848 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.968772    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:14.968848 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.968840    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:15.968821067 +0000 UTC m=+3.956042596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:14.969035 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.968772    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:14.969035 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.968904    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret podName:e0a25e8a-583a-4194-a8df-ff565cac7a43 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:15.968892429 +0000 UTC m=+3.956113949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret") pod "global-pull-secret-syncer-pwtgz" (UID: "e0a25e8a-583a-4194-a8df-ff565cac7a43") : object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:14.969035 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.968776    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 12:15:14.969035 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.968923    2586 projected.go:196] Error preparing data for projected volume kube-api-access-n5z67 for pod openshift-network-diagnostics/network-check-target-kh24k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:14.969035 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:14.968950    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67 podName:e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c nodeName:}" failed. No retries permitted until 2026-07-31 12:15:15.968941378 +0000 UTC m=+3.956162908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5z67" (UniqueName: "kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67") pod "network-check-target-kh24k" (UID: "e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:15.392687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.392645    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 12:10:14 +0000 UTC" deadline="2028-03-12 18:14:07.22375431 +0000 UTC"
Jul 31 12:15:15.392687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.392683    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14165h58m51.831073839s"
Jul 31 12:15:15.420441 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:15.420389    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd73de9_7b12_4de5_ab9b_875c7bf9d62c.slice/crio-6b9ce0584612b6159179a6eafa821e8653074a9c59e0548a63e97d258f462ccc WatchSource:0}: Error finding container 6b9ce0584612b6159179a6eafa821e8653074a9c59e0548a63e97d258f462ccc: Status 404 returned error can't find the container with id 6b9ce0584612b6159179a6eafa821e8653074a9c59e0548a63e97d258f462ccc
Jul 31 12:15:15.422094 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:15.422068    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b1fc55_23f2_41da_981d_51d4f737c4e5.slice/crio-9552c3a63cebc0dfb66ef29e5076fa562e01ae236b1ba3cfedd1b8adb00ee178 WatchSource:0}: Error finding container 9552c3a63cebc0dfb66ef29e5076fa562e01ae236b1ba3cfedd1b8adb00ee178: Status 404 returned error can't find the container with id 9552c3a63cebc0dfb66ef29e5076fa562e01ae236b1ba3cfedd1b8adb00ee178
Jul 31 12:15:15.423325 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:15.423303    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827b42e8_32dd_4f96_93ee_fbbad611cc5f.slice/crio-328ba8552ff668660baa031bd24c5a725994853d06c98b8cc7ef23304d067182 WatchSource:0}: Error finding container 328ba8552ff668660baa031bd24c5a725994853d06c98b8cc7ef23304d067182: Status 404 returned error can't find the container with id 328ba8552ff668660baa031bd24c5a725994853d06c98b8cc7ef23304d067182
Jul 31 12:15:15.425992 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:15.425968    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c4f51d_6014_4c7c_8562_034f42cbf93e.slice/crio-8492b1913a09548def38cef15eeca4167ddc265be9fbdbdb59c165c7a88d86df WatchSource:0}: Error finding container 8492b1913a09548def38cef15eeca4167ddc265be9fbdbdb59c165c7a88d86df: Status 404 returned error can't find the container with id 8492b1913a09548def38cef15eeca4167ddc265be9fbdbdb59c165c7a88d86df
Jul 31 12:15:15.426508 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:15.426488    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd89af6f_0f7c_4bef_9a41_5e8f58c5d0f7.slice/crio-96fe71395341735274907b661fe649423f5fe7394f11eff8341a8b8de7399982 WatchSource:0}: Error finding container 96fe71395341735274907b661fe649423f5fe7394f11eff8341a8b8de7399982: Status 404 returned error can't find the container with id 96fe71395341735274907b661fe649423f5fe7394f11eff8341a8b8de7399982
Jul 31 12:15:15.427441 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:15.427315    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb11a5be_943c_4a5c_8ce3_06b1fab1c840.slice/crio-22d08a72e45558e594598adefa9d3fb3a9e2c77347fbce8e5698c033bf63de47 WatchSource:0}: Error finding container 22d08a72e45558e594598adefa9d3fb3a9e2c77347fbce8e5698c033bf63de47: Status 404 returned error can't find the container with id 22d08a72e45558e594598adefa9d3fb3a9e2c77347fbce8e5698c033bf63de47
Jul 31 12:15:15.478503 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.478377    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:15.478622 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:15.478597    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:15.488160 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.488139    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" event={"ID":"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223","Type":"ContainerStarted","Data":"8ca99c601b5d586d3f7ac958f0454afa4bf32a9468608b5ff84f49a301609bdd"}
Jul 31 12:15:15.489275 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.489236    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6" event={"ID":"9009cfca-fe02-47d8-a473-75e9d6b0cd8e","Type":"ContainerStarted","Data":"94a798a07c104f14c2d33b9fc5b91001a1fab909670f40dadd4c37eca4f5984b"}
Jul 31 12:15:15.490191 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.490174    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lhqd7" event={"ID":"eb11a5be-943c-4a5c-8ce3-06b1fab1c840","Type":"ContainerStarted","Data":"22d08a72e45558e594598adefa9d3fb3a9e2c77347fbce8e5698c033bf63de47"}
Jul 31 12:15:15.491157 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.491127    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jsrdj" event={"ID":"27c4f51d-6014-4c7c-8562-034f42cbf93e","Type":"ContainerStarted","Data":"8492b1913a09548def38cef15eeca4167ddc265be9fbdbdb59c165c7a88d86df"}
Jul 31 12:15:15.492086 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.492054    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wh6xs" event={"ID":"827b42e8-32dd-4f96-93ee-fbbad611cc5f","Type":"ContainerStarted","Data":"328ba8552ff668660baa031bd24c5a725994853d06c98b8cc7ef23304d067182"}
Jul 31 12:15:15.493095 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.493074    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jl8fm" event={"ID":"70b1fc55-23f2-41da-981d-51d4f737c4e5","Type":"ContainerStarted","Data":"9552c3a63cebc0dfb66ef29e5076fa562e01ae236b1ba3cfedd1b8adb00ee178"}
Jul 31 12:15:15.494277 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.494240    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-plkjn" event={"ID":"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c","Type":"ContainerStarted","Data":"6b9ce0584612b6159179a6eafa821e8653074a9c59e0548a63e97d258f462ccc"}
Jul 31 12:15:15.495180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.495163    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8sxf" event={"ID":"b1074883-7ab6-46a5-a22c-07e17d691672","Type":"ContainerStarted","Data":"367fd67397087737fcda2608309c2917bfe2a9056655fca86d8ca12dd41f003d"}
Jul 31 12:15:15.497077 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.497059    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdb4h" event={"ID":"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7","Type":"ContainerStarted","Data":"96fe71395341735274907b661fe649423f5fe7394f11eff8341a8b8de7399982"}
Jul 31 12:15:15.976543 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.976511    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:15.976687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.976585    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:15.976687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:15.976612    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:15.976793 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:15.976748    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:15.976838 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:15.976817    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:17.976800401 +0000 UTC m=+5.964021930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:15.977208 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:15.977190    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:15.977331 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:15.977280    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret podName:e0a25e8a-583a-4194-a8df-ff565cac7a43 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:17.977246477 +0000 UTC m=+5.964467998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret") pod "global-pull-secret-syncer-pwtgz" (UID: "e0a25e8a-583a-4194-a8df-ff565cac7a43") : object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:15.977397 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:15.977365    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 12:15:15.977397 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:15.977386    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 12:15:15.977397 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:15.977397    2586 projected.go:196] Error preparing data for projected volume kube-api-access-n5z67 for pod openshift-network-diagnostics/network-check-target-kh24k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:15.977530 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:15.977452    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67 podName:e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c nodeName:}" failed. No retries permitted until 2026-07-31 12:15:17.977422313 +0000 UTC m=+5.964643834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5z67" (UniqueName: "kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67") pod "network-check-target-kh24k" (UID: "e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:16.352691 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:16.352616    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 12:15:16.393346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:16.393219    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 12:10:14 +0000 UTC" deadline="2028-03-21 19:54:36.840436275 +0000 UTC"
Jul 31 12:15:16.393346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:16.393249    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14383h39m20.447195767s"
Jul 31 12:15:16.481459 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:16.480725    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:16.481459 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:16.480882    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:16.481459 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:16.481285    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:16.481459 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:16.481395    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:16.520910 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:16.520140    2586 generic.go:356] "Generic (PLEG): container finished" podID="8ee029fa264a2a647e1d318de5a3789c" containerID="0c3f9fb06000fdbf0a350f05c9e2234ab8c745b5b3f97e25feeeb209dc192071" exitCode=0
Jul 31 12:15:16.520910 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:16.520241    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" event={"ID":"8ee029fa264a2a647e1d318de5a3789c","Type":"ContainerDied","Data":"0c3f9fb06000fdbf0a350f05c9e2234ab8c745b5b3f97e25feeeb209dc192071"}
Jul 31 12:15:16.541118 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:16.541073    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" event={"ID":"d64eb269ad67e5f98e8a744fc0f2cae5","Type":"ContainerStarted","Data":"7517fb5aa3355fa51047239b9073527d8f2073d0977f41116bc6ea3b290aa95d"}
Jul 31 12:15:17.479221 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:17.479190    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:17.479713 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:17.479332    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:17.560393 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:17.560323    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" event={"ID":"8ee029fa264a2a647e1d318de5a3789c","Type":"ContainerStarted","Data":"62992047983d927425c82df1503e3e6f9de632d5d3c1885e8aa5d34320de1b5d"}
Jul 31 12:15:17.576999 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:17.576098    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" podStartSLOduration=4.576076861 podStartE2EDuration="4.576076861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:17.57561795 +0000 UTC m=+5.562839490" watchObservedRunningTime="2026-07-31 12:15:17.576076861 +0000 UTC m=+5.563298402"
Jul 31 12:15:17.576999 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:17.576446    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" podStartSLOduration=4.576437275 podStartE2EDuration="4.576437275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:16.560912147 +0000 UTC m=+4.548133686" watchObservedRunningTime="2026-07-31 12:15:17.576437275 +0000 UTC m=+5.563658815"
Jul 31 12:15:17.993568 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:17.993532    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:17.993754 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:17.993587    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:17.993754 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:17.993639    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:17.993831 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:17.993801    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:17.993899 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:17.993863    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret podName:e0a25e8a-583a-4194-a8df-ff565cac7a43 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:21.993845811 +0000 UTC m=+9.981067346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret") pod "global-pull-secret-syncer-pwtgz" (UID: "e0a25e8a-583a-4194-a8df-ff565cac7a43") : object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:17.994309 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:17.994291    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 12:15:17.994408 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:17.994317    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 12:15:17.994408 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:17.994331    2586 projected.go:196] Error preparing data for projected volume kube-api-access-n5z67 for pod openshift-network-diagnostics/network-check-target-kh24k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:17.994408 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:17.994375    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67 podName:e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c nodeName:}" failed. No retries permitted until 2026-07-31 12:15:21.994359714 +0000 UTC m=+9.981581249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5z67" (UniqueName: "kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67") pod "network-check-target-kh24k" (UID: "e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:17.994553 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:17.994441    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:17.994553 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:17.994475    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:21.994461554 +0000 UTC m=+9.981683073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:18.480367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:18.479872    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:18.480367 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:18.480008    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:18.480882 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:18.480406    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:18.480882 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:18.480540    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:19.478537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:19.478505    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:19.478675 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:19.478619    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:20.478655 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:20.478625    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:20.479003 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:20.478757    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:20.479467 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:20.478596    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:20.479467 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:20.479419    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:21.479084 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:21.479051    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:21.479607 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:21.479180    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:22.025712    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:22.025771    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:22.025812    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.025904    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.025930    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.025935    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.025960    2586 projected.go:196] Error preparing data for projected volume kube-api-access-n5z67 for pod openshift-network-diagnostics/network-check-target-kh24k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.025991    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret podName:e0a25e8a-583a-4194-a8df-ff565cac7a43 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:30.025973886 +0000 UTC m=+18.013195407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret") pod "global-pull-secret-syncer-pwtgz" (UID: "e0a25e8a-583a-4194-a8df-ff565cac7a43") : object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.025931    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.026010    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67 podName:e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c nodeName:}" failed. No retries permitted until 2026-07-31 12:15:30.026000996 +0000 UTC m=+18.013222520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5z67" (UniqueName: "kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67") pod "network-check-target-kh24k" (UID: "e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:22.026018 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.026036    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:30.026019033 +0000 UTC m=+18.013240564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:22.480292 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:22.478529    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:22.480292 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.478660    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:22.480292 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:22.479067    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:22.480292 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:22.479168    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:23.478719 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:23.478685    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:23.478891 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:23.478823    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:24.479230 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:24.479119    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:24.479724 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:24.479258    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:24.479724 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:24.479299    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:24.479724 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:24.479425    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:25.478702 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:25.478673    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:25.478796 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:25.478778    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:26.478803 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:26.478763    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:26.479250 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:26.478781    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:26.479250 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:26.478911    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:26.479250 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:26.479010    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:27.479378 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:27.479339    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:27.479838 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:27.479470    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:28.479391 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:28.479354    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:28.479838 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:28.479359    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:28.479838 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:28.479495    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:28.479838 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:28.479557    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:29.478804 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:29.478779    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:29.478982 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:29.478879    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:30.091915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:30.091863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:30.091915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:30.091917    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:30.092399 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:30.091955    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:30.092399 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.092049    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:30.092399 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.092061    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:30.092399 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.092051    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 12:15:30.092399 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.092110    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:46.09209496 +0000 UTC m=+34.079316494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:30.092399 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.092122    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 12:15:30.092399 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.092135    2586 projected.go:196] Error preparing data for projected volume kube-api-access-n5z67 for pod openshift-network-diagnostics/network-check-target-kh24k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:30.092399 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.092125    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret podName:e0a25e8a-583a-4194-a8df-ff565cac7a43 nodeName:}" failed. No retries permitted until 2026-07-31 12:15:46.092119243 +0000 UTC m=+34.079340761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret") pod "global-pull-secret-syncer-pwtgz" (UID: "e0a25e8a-583a-4194-a8df-ff565cac7a43") : object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:30.092399 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.092183    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67 podName:e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c nodeName:}" failed. No retries permitted until 2026-07-31 12:15:46.092168459 +0000 UTC m=+34.079389983 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5z67" (UniqueName: "kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67") pod "network-check-target-kh24k" (UID: "e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:30.478539 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:30.478507    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:30.478776 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.478635    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:30.478776 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:30.478682    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:30.478892 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:30.478777    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:31.478592 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:31.478554    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:31.478968 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:31.478672    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:32.479999 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:32.479957    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:32.480410 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:32.480070    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:32.480410 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:32.480162    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:32.480410 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:32.480300    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:33.480226 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:33.479163    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:33.480226 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:33.479378    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:33.597652 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:33.597505    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wh6xs" event={"ID":"827b42e8-32dd-4f96-93ee-fbbad611cc5f","Type":"ContainerStarted","Data":"fcbe1eb2f81767761e0a78c5b5a78bdb3eacfad7fbbdce4181d549558aa9d4f8"}
Jul 31 12:15:33.617247 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:33.617190    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wh6xs" podStartSLOduration=3.6056807600000003 podStartE2EDuration="21.617168172s" totalImagesPullingTime="18.011487412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:33.616579975 +0000 UTC m=+21.603801512" watchObservedRunningTime="2026-07-31 12:15:33.617168172 +0000 UTC m=+21.604389713"
Jul 31 12:15:34.479436 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.479277    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:34.479568 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.479336    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:34.479568 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:34.479521    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:34.479660 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:34.479584    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:34.601522 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.601482    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" event={"ID":"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223","Type":"ContainerStarted","Data":"3a481e6db812a5e7f2da1974e98840ee93190092403e351df1abc9bda76d31bc"}
Jul 31 12:15:34.602176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.601580    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" event={"ID":"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223","Type":"ContainerStarted","Data":"48f8660e5db2f0e2abf8624f9be3f736221f6be5735a682e7ef18ef3f71ad540"}
Jul 31 12:15:34.602176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.601632    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" event={"ID":"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223","Type":"ContainerStarted","Data":"4dd9b925381eaf80e971dc012d94229142925afb39798984bb41c3c3821a9907"}
Jul 31 12:15:34.602176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.601677    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" event={"ID":"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223","Type":"ContainerStarted","Data":"2afbdf4e3aa07175cb43fe7cdd2b75210d8f9138ad47f8d32aa284b5285509f4"}
Jul 31 12:15:34.602176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.601774    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" event={"ID":"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223","Type":"ContainerStarted","Data":"bd4170193cd3aeb6e72b7656685c0696913436e53b3bdf9d6b58e94976b243d6"}
Jul 31 12:15:34.602176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.601821    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" event={"ID":"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223","Type":"ContainerStarted","Data":"49de8d31a2854ff56e1ac1864d9c2affde0d3b24b48a42265bdb11e323a5c18e"}
Jul 31 12:15:34.602841 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.602822    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6" event={"ID":"9009cfca-fe02-47d8-a473-75e9d6b0cd8e","Type":"ContainerStarted","Data":"c47e81b9fd1f032509e8d09c96d72f29c71d9495ea507cfde88317ce1cfa7940"}
Jul 31 12:15:34.604167 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.604145    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lhqd7" event={"ID":"eb11a5be-943c-4a5c-8ce3-06b1fab1c840","Type":"ContainerStarted","Data":"f03f3b8fcd9dfb414d36dc12b4fa5d5beb6a6b47c35c0260151706a94784426e"}
Jul 31 12:15:34.605480 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.605455    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jsrdj" event={"ID":"27c4f51d-6014-4c7c-8562-034f42cbf93e","Type":"ContainerStarted","Data":"3b5c43e690309a0e3b898291a6dd9106aee0b5319643e4f428678884386bcb9c"}
Jul 31 12:15:34.606808 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.606786    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jl8fm" event={"ID":"70b1fc55-23f2-41da-981d-51d4f737c4e5","Type":"ContainerStarted","Data":"8fb4787dd208fd751ff812c4cebcf8093f09c3a39c9b0d4b6e770e2bd863f565"}
Jul 31 12:15:34.608034 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.608013    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-plkjn" event={"ID":"bfd73de9-7b12-4de5-ab9b-875c7bf9d62c","Type":"ContainerStarted","Data":"b85f7670301b31cb3ddcdcf608214ed9494439d7032f75bad984f4016cd49269"}
Jul 31 12:15:34.609382 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.609353    2586 generic.go:356] "Generic (PLEG): container finished" podID="b1074883-7ab6-46a5-a22c-07e17d691672" containerID="88312f7fc05902d3e760dc9f7d58f5b65325a82f58eed87a3a8bb70778f7a9e1" exitCode=0
Jul 31 12:15:34.609459 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.609415    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8sxf" event={"ID":"b1074883-7ab6-46a5-a22c-07e17d691672","Type":"ContainerDied","Data":"88312f7fc05902d3e760dc9f7d58f5b65325a82f58eed87a3a8bb70778f7a9e1"}
Jul 31 12:15:34.620867 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.620828    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lhqd7" podStartSLOduration=4.561191954 podStartE2EDuration="22.62081587s" totalImagesPullingTime="18.059623916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:34.620739579 +0000 UTC m=+22.607961119" watchObservedRunningTime="2026-07-31 12:15:34.62081587 +0000 UTC m=+22.608037409"
Jul 31 12:15:34.638377 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.638340    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jsrdj" podStartSLOduration=8.801740819 podStartE2EDuration="22.638329491s" totalImagesPullingTime="13.836588672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:34.637872154 +0000 UTC m=+22.625093688" watchObservedRunningTime="2026-07-31 12:15:34.638329491 +0000 UTC m=+22.625551030"
Jul 31 12:15:34.651414 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.651381    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-plkjn" podStartSLOduration=4.639044039 podStartE2EDuration="22.651370001s" totalImagesPullingTime="18.012325962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:34.650864979 +0000 UTC m=+22.638086518" watchObservedRunningTime="2026-07-31 12:15:34.651370001 +0000 UTC m=+22.638591539"
Jul 31 12:15:34.666004 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:34.665974    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jl8fm" podStartSLOduration=4.655265796 podStartE2EDuration="22.665962946s" totalImagesPullingTime="18.01069715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:34.665793175 +0000 UTC m=+22.653014718" watchObservedRunningTime="2026-07-31 12:15:34.665962946 +0000 UTC m=+22.653184485"
Jul 31 12:15:35.305061 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.305024    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 12:15:35.425477 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.425345    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T12:15:35.30504001Z","UUID":"a6e9f902-4798-49dd-84b9-21a7f04ae398","Handler":null,"Name":"","Endpoint":""}
Jul 31 12:15:35.426739 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.426721    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 12:15:35.426849 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.426759    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 12:15:35.479134 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.479111    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:35.479346 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:35.479216    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:35.613783 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.613749    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdb4h" event={"ID":"bd89af6f-0f7c-4bef-9a41-5e8f58c5d0f7","Type":"ContainerStarted","Data":"17f53cc4fb7978a531c00a8d973adda738ed501f4593cd8b4ba0089bf803dec3"}
Jul 31 12:15:35.615873 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.615828    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6" event={"ID":"9009cfca-fe02-47d8-a473-75e9d6b0cd8e","Type":"ContainerStarted","Data":"b1e7aae640fd6c53e2aee20c0fbea9fdbc6263a707e1c812fef17c3e9e67173b"}
Jul 31 12:15:35.628860 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.628808    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zdb4h" podStartSLOduration=13.568446805 podStartE2EDuration="23.628789425s" totalImagesPullingTime="10.06034262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:35.62801533 +0000 UTC m=+23.615236872" watchObservedRunningTime="2026-07-31 12:15:35.628789425 +0000 UTC m=+23.616010967"
Jul 31 12:15:35.648159 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.648133    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:35.648785 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:35.648743    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:36.479512 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:36.479313    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:36.479738 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:36.479359    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:36.479738 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:36.479621    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:36.479738 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:36.479711    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:36.622422 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:36.622326    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" event={"ID":"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223","Type":"ContainerStarted","Data":"485dc5aeec790ad4b928bdbb002d77c235c21fbdecdd3017e9f5cd0b11a8e8d4"}
Jul 31 12:15:36.622878 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:36.622719    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:36.623156 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:36.623139    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jsrdj"
Jul 31 12:15:37.478801 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:37.478774    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:37.479069 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:37.478877    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:37.626513 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:37.626477    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6" event={"ID":"9009cfca-fe02-47d8-a473-75e9d6b0cd8e","Type":"ContainerStarted","Data":"0fc18941d4be912d4c0cea95216e5e3199a99cab09bbde62af707f2c22da9cb6"}
Jul 31 12:15:37.647015 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:37.646971    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vt4v6" podStartSLOduration=4.7282498870000005 podStartE2EDuration="25.646956311s" totalImagesPullingTime="20.918706424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:37.646757337 +0000 UTC m=+25.633978878" watchObservedRunningTime="2026-07-31 12:15:37.646956311 +0000 UTC m=+25.634177851"
Jul 31 12:15:38.479173 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:38.479138    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:38.479369 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:38.479246    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:38.479369 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:38.479328    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:38.479495 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:38.479440    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:39.479085 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:39.479053    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:39.479643 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:39.479186    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:40.481712 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:40.481542    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:40.482354 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:40.481595    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:40.482354 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:40.481808    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:40.482354 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:40.481839    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:40.634352 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:40.634258    2586 generic.go:356] "Generic (PLEG): container finished" podID="b1074883-7ab6-46a5-a22c-07e17d691672" containerID="c0d76c203d640e3407c9338da3000b4dd9bb1ef35c8512630e3e1f168bf24376" exitCode=0
Jul 31 12:15:40.634541 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:40.634360    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8sxf" event={"ID":"b1074883-7ab6-46a5-a22c-07e17d691672","Type":"ContainerDied","Data":"c0d76c203d640e3407c9338da3000b4dd9bb1ef35c8512630e3e1f168bf24376"}
Jul 31 12:15:40.638200 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:40.638180    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" event={"ID":"c2bd1d2d-cc6f-4dcb-84d6-70b8f0c38223","Type":"ContainerStarted","Data":"aeb885423f298d2e443d5cc960b2e2093bd4992f4979d38575872104264b0c76"}
Jul 31 12:15:40.638448 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:40.638430    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:40.638510 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:40.638504    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:40.653462 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:40.653443    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:40.679055 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:40.679018    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d" podStartSLOduration=10.593753797 podStartE2EDuration="28.679005895s" totalImagesPullingTime="18.085252098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:40.678739156 +0000 UTC m=+28.665960696" watchObservedRunningTime="2026-07-31 12:15:40.679005895 +0000 UTC m=+28.666227435"
Jul 31 12:15:41.479196 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:41.479171    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:41.479475 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:41.479304    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:41.640570 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:41.640543    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:41.654980 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:41.654961    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:15:41.704406 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:41.704382    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pwtgz"]
Jul 31 12:15:41.704546 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:41.704533    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:41.704632 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:41.704615    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:41.708213 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:41.708186    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kh24k"]
Jul 31 12:15:41.708398 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:41.708384    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:41.708528 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:41.708498    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:41.708961 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:41.708940    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cf94v"]
Jul 31 12:15:41.709097 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:41.709046    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:41.709160 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:41.709143    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:42.644340 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:42.644107    2586 generic.go:356] "Generic (PLEG): container finished" podID="b1074883-7ab6-46a5-a22c-07e17d691672" containerID="dcc864a0b34d6b64f8c8731d9a5aeb272ceb88800cec89a57df91cd7d86856b7" exitCode=0
Jul 31 12:15:42.644688 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:42.644178    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8sxf" event={"ID":"b1074883-7ab6-46a5-a22c-07e17d691672","Type":"ContainerDied","Data":"dcc864a0b34d6b64f8c8731d9a5aeb272ceb88800cec89a57df91cd7d86856b7"}
Jul 31 12:15:43.479114 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:43.479089    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:43.479260 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:43.479095    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:43.479260 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:43.479189    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:43.479260 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:43.479205    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:43.479411 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:43.479297    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:43.479411 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:43.479375    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:44.650663 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:44.650626    2586 generic.go:356] "Generic (PLEG): container finished" podID="b1074883-7ab6-46a5-a22c-07e17d691672" containerID="f0cdec9d4d384376b88700178ab97a7629e9fc949d9f7712a9de0c08e81e76e5" exitCode=0
Jul 31 12:15:44.651379 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:44.650694    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8sxf" event={"ID":"b1074883-7ab6-46a5-a22c-07e17d691672","Type":"ContainerDied","Data":"f0cdec9d4d384376b88700178ab97a7629e9fc949d9f7712a9de0c08e81e76e5"}
Jul 31 12:15:45.478616 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:45.478566    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:45.478851 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:45.478583    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:45.478851 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:45.478697    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh24k" podUID="e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c"
Jul 31 12:15:45.478851 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:45.478707    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:45.478851 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:45.478798    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pwtgz" podUID="e0a25e8a-583a-4194-a8df-ff565cac7a43"
Jul 31 12:15:45.479031 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:45.478937    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:15:46.118059 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:46.118020    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:46.118059 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:46.118057    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:46.118680 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:46.118084    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:46.118680 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:46.118199    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:46.118680 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:46.118207    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 12:15:46.118680 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:46.118228    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 12:15:46.118680 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:46.118237    2586 projected.go:196] Error preparing data for projected volume kube-api-access-n5z67 for pod openshift-network-diagnostics/network-check-target-kh24k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:46.118680 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:46.118250    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret podName:e0a25e8a-583a-4194-a8df-ff565cac7a43 nodeName:}" failed. No retries permitted until 2026-07-31 12:16:18.118237534 +0000 UTC m=+66.105459052 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret") pod "global-pull-secret-syncer-pwtgz" (UID: "e0a25e8a-583a-4194-a8df-ff565cac7a43") : object "kube-system"/"original-pull-secret" not registered
Jul 31 12:15:46.118680 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:46.118199    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:46.118680 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:46.118294    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67 podName:e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c nodeName:}" failed. No retries permitted until 2026-07-31 12:16:18.118278931 +0000 UTC m=+66.105500463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5z67" (UniqueName: "kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67") pod "network-check-target-kh24k" (UID: "e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 12:15:46.118680 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:46.118313    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:16:18.11830627 +0000 UTC m=+66.105527787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 12:15:47.318844 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.318773    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeReady"
Jul 31 12:15:47.319161 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.318886    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 12:15:47.366914 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.366883    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9d7l8"]
Jul 31 12:15:47.369504 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.369477    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.371998 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.371915    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2qljl"]
Jul 31 12:15:47.372372 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.372325    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ngjpv\""
Jul 31 12:15:47.372456 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.372427    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 12:15:47.372712 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.372689    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 12:15:47.373819 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.373795    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2qljl"
Jul 31 12:15:47.376207 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.376190    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4h696\""
Jul 31 12:15:47.376472 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.376455    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 12:15:47.376559 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.376456    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 12:15:47.376606 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.376457    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 12:15:47.381983 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.381958    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9d7l8"]
Jul 31 12:15:47.384838 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.384790    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2qljl"]
Jul 31 12:15:47.479422 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.479391    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:15:47.479588 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.479391    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:15:47.479644 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.479391    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:15:47.482746 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.482711    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 12:15:47.482918 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.482904    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 12:15:47.482988 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.482974    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 12:15:47.483087 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.483073    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 12:15:47.483170 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.483155    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qqfkp\""
Jul 31 12:15:47.484311 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.484292    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dw9nv\""
Jul 31 12:15:47.527934 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.527906    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-config-volume\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.528043 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.527953    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqnj\" (UniqueName: \"kubernetes.io/projected/7b7772db-0980-4ddb-ade7-cf9c5ec44a40-kube-api-access-hhqnj\") pod \"ingress-canary-2qljl\" (UID: \"7b7772db-0980-4ddb-ade7-cf9c5ec44a40\") " pod="openshift-ingress-canary/ingress-canary-2qljl"
Jul 31 12:15:47.528043 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.527988    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.528043 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.528012    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-tmp-dir\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.528196 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.528134    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmn4m\" (UniqueName: \"kubernetes.io/projected/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-kube-api-access-mmn4m\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.528240 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.528191    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b7772db-0980-4ddb-ade7-cf9c5ec44a40-cert\") pod \"ingress-canary-2qljl\" (UID: \"7b7772db-0980-4ddb-ade7-cf9c5ec44a40\") " pod="openshift-ingress-canary/ingress-canary-2qljl"
Jul 31 12:15:47.629352 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.629253    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqnj\" (UniqueName: \"kubernetes.io/projected/7b7772db-0980-4ddb-ade7-cf9c5ec44a40-kube-api-access-hhqnj\") pod \"ingress-canary-2qljl\" (UID: \"7b7772db-0980-4ddb-ade7-cf9c5ec44a40\") " pod="openshift-ingress-canary/ingress-canary-2qljl"
Jul 31 12:15:47.629352 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.629313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.629352 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.629333    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-tmp-dir\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.629614 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.629416    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmn4m\" (UniqueName: \"kubernetes.io/projected/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-kube-api-access-mmn4m\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.629614 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:47.629434    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 12:15:47.629614 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.629455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b7772db-0980-4ddb-ade7-cf9c5ec44a40-cert\") pod \"ingress-canary-2qljl\" (UID: \"7b7772db-0980-4ddb-ade7-cf9c5ec44a40\") " pod="openshift-ingress-canary/ingress-canary-2qljl"
Jul 31 12:15:47.629614 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.629490    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-config-volume\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.629732 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:47.629637    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls podName:6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa nodeName:}" failed. No retries permitted until 2026-07-31 12:15:48.129614746 +0000 UTC m=+36.116836264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls") pod "dns-default-9d7l8" (UID: "6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa") : secret "dns-default-metrics-tls" not found
Jul 31 12:15:47.629772 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.629730    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-tmp-dir\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.630134 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.630116    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-config-volume\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.634217 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.634192    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b7772db-0980-4ddb-ade7-cf9c5ec44a40-cert\") pod \"ingress-canary-2qljl\" (UID: \"7b7772db-0980-4ddb-ade7-cf9c5ec44a40\") " pod="openshift-ingress-canary/ingress-canary-2qljl"
Jul 31 12:15:47.640107 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.640082    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmn4m\" (UniqueName: \"kubernetes.io/projected/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-kube-api-access-mmn4m\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:47.640648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.640626    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqnj\" (UniqueName: \"kubernetes.io/projected/7b7772db-0980-4ddb-ade7-cf9c5ec44a40-kube-api-access-hhqnj\") pod \"ingress-canary-2qljl\" (UID: \"7b7772db-0980-4ddb-ade7-cf9c5ec44a40\") " pod="openshift-ingress-canary/ingress-canary-2qljl"
Jul 31 12:15:47.688010 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:47.687990    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2qljl"
Jul 31 12:15:48.134674 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:48.134432    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:48.134842 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:48.134575    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 12:15:48.134842 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:48.134821    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls podName:6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa nodeName:}" failed. No retries permitted until 2026-07-31 12:15:49.134799807 +0000 UTC m=+37.122021340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls") pod "dns-default-9d7l8" (UID: "6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa") : secret "dns-default-metrics-tls" not found
Jul 31 12:15:48.944534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:48.944497    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2qljl"]
Jul 31 12:15:48.948285 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:15:48.948238    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7772db_0980_4ddb_ade7_cf9c5ec44a40.slice/crio-331051dcdd5c07743111020d66435d60988319cb875fe3cff5de3861032a513f WatchSource:0}: Error finding container 331051dcdd5c07743111020d66435d60988319cb875fe3cff5de3861032a513f: Status 404 returned error can't find the container with id 331051dcdd5c07743111020d66435d60988319cb875fe3cff5de3861032a513f
Jul 31 12:15:49.144234 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:49.144112    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:49.144393 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:49.144296    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 12:15:49.144393 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:49.144370    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls podName:6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa nodeName:}" failed. No retries permitted until 2026-07-31 12:15:51.144355421 +0000 UTC m=+39.131576938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls") pod "dns-default-9d7l8" (UID: "6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa") : secret "dns-default-metrics-tls" not found
Jul 31 12:15:49.663954 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:49.663913    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2qljl" event={"ID":"7b7772db-0980-4ddb-ade7-cf9c5ec44a40","Type":"ContainerStarted","Data":"331051dcdd5c07743111020d66435d60988319cb875fe3cff5de3861032a513f"}
Jul 31 12:15:49.667069 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:49.667043    2586 generic.go:356] "Generic (PLEG): container finished" podID="b1074883-7ab6-46a5-a22c-07e17d691672" containerID="49f05fcf77c141703b7495d54b9e0c8f41d05e90875366f311cf5f7e5f3f7516" exitCode=0
Jul 31 12:15:49.667189 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:49.667078    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8sxf" event={"ID":"b1074883-7ab6-46a5-a22c-07e17d691672","Type":"ContainerDied","Data":"49f05fcf77c141703b7495d54b9e0c8f41d05e90875366f311cf5f7e5f3f7516"}
Jul 31 12:15:50.673280 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:50.673038    2586 generic.go:356] "Generic (PLEG): container finished" podID="b1074883-7ab6-46a5-a22c-07e17d691672" containerID="76c0cb3be8bb31668e67539d5e9eae8e679a0e9779703d61c4b007ca8c2b0601" exitCode=0
Jul 31 12:15:50.673735 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:50.673116    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8sxf" event={"ID":"b1074883-7ab6-46a5-a22c-07e17d691672","Type":"ContainerDied","Data":"76c0cb3be8bb31668e67539d5e9eae8e679a0e9779703d61c4b007ca8c2b0601"}
Jul 31 12:15:51.157504 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:51.157478    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:51.157654 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:51.157609    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 12:15:51.157691 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:51.157661    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls podName:6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa nodeName:}" failed. No retries permitted until 2026-07-31 12:15:55.157647023 +0000 UTC m=+43.144868541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls") pod "dns-default-9d7l8" (UID: "6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa") : secret "dns-default-metrics-tls" not found
Jul 31 12:15:51.677166 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:51.677127    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2qljl" event={"ID":"7b7772db-0980-4ddb-ade7-cf9c5ec44a40","Type":"ContainerStarted","Data":"8aaf07e7f3bf4b0c3dbcc191d07ea8e71453b924ac84172b06ac3f11e74e2958"}
Jul 31 12:15:51.680535 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:51.680510    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8sxf" event={"ID":"b1074883-7ab6-46a5-a22c-07e17d691672","Type":"ContainerStarted","Data":"416d46f3bddb9ee44201a53385f2a420bc3871ffab7f67bf0881ae2449e6815f"}
Jul 31 12:15:51.692146 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:51.692100    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2qljl" podStartSLOduration=2.1761545509999998 podStartE2EDuration="4.692083759s" totalImagesPullingTime="2.515929208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:51.691133195 +0000 UTC m=+39.678354795" watchObservedRunningTime="2026-07-31 12:15:51.692083759 +0000 UTC m=+39.679305353"
Jul 31 12:15:51.714568 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:51.714524    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k8sxf" podStartSLOduration=10.314085968 podStartE2EDuration="39.714509385s" totalImagesPullingTime="29.400423417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:15:51.713364453 +0000 UTC m=+39.700585988" watchObservedRunningTime="2026-07-31 12:15:51.714509385 +0000 UTC m=+39.701730921"
Jul 31 12:15:55.188485 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:15:55.188450    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:15:55.188830 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:55.188602    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 12:15:55.188830 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:15:55.188672    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls podName:6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa nodeName:}" failed. No retries permitted until 2026-07-31 12:16:03.188653954 +0000 UTC m=+51.175875494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls") pod "dns-default-9d7l8" (UID: "6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa") : secret "dns-default-metrics-tls" not found
Jul 31 12:16:03.246632 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:03.246590    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:16:03.247108 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:16:03.246777    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 12:16:03.247108 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:16:03.246853    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls podName:6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa nodeName:}" failed. No retries permitted until 2026-07-31 12:16:19.246836449 +0000 UTC m=+67.234057966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls") pod "dns-default-9d7l8" (UID: "6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa") : secret "dns-default-metrics-tls" not found
Jul 31 12:16:13.656585 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:13.656556    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rv4d"
Jul 31 12:16:18.152447 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.152393    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:16:18.152447 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.152450    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:16:18.152984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.152478    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:16:18.156313 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.156294    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 12:16:18.156383 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.156323    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 12:16:18.156419 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.156330    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 12:16:18.163277 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:16:18.163248    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 12:16:18.163386 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:16:18.163349    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:17:22.163328299 +0000 UTC m=+130.150549835 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : secret "metrics-daemon-secret" not found
Jul 31 12:16:18.165129 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.165115    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 12:16:18.166897 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.166878    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0a25e8a-583a-4194-a8df-ff565cac7a43-original-pull-secret\") pod \"global-pull-secret-syncer-pwtgz\" (UID: \"e0a25e8a-583a-4194-a8df-ff565cac7a43\") " pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:16:18.175812 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.175785    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5z67\" (UniqueName: \"kubernetes.io/projected/e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c-kube-api-access-n5z67\") pod \"network-check-target-kh24k\" (UID: \"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c\") " pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:16:18.393813 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.393782    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dw9nv\""
Jul 31 12:16:18.401998 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.401978    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:16:18.403248 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.403205    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pwtgz"
Jul 31 12:16:18.546929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.546901    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kh24k"]
Jul 31 12:16:18.550482 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:16:18.550455    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode729cbc6_e3c4_491f_9f6a_4b843eeb5b1c.slice/crio-1370503c4cca05a7e3f22d16d846ce874572b7c81098c173565e718f176d3ec9 WatchSource:0}: Error finding container 1370503c4cca05a7e3f22d16d846ce874572b7c81098c173565e718f176d3ec9: Status 404 returned error can't find the container with id 1370503c4cca05a7e3f22d16d846ce874572b7c81098c173565e718f176d3ec9
Jul 31 12:16:18.561315 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.561293    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pwtgz"]
Jul 31 12:16:18.564653 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:16:18.564636    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a25e8a_583a_4194_a8df_ff565cac7a43.slice/crio-f68bcec7de970c651fcef43ebe881c4dd69284295e23553a802fdb04c9fab94d WatchSource:0}: Error finding container f68bcec7de970c651fcef43ebe881c4dd69284295e23553a802fdb04c9fab94d: Status 404 returned error can't find the container with id f68bcec7de970c651fcef43ebe881c4dd69284295e23553a802fdb04c9fab94d
Jul 31 12:16:18.741103 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.741021    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pwtgz" event={"ID":"e0a25e8a-583a-4194-a8df-ff565cac7a43","Type":"ContainerStarted","Data":"f68bcec7de970c651fcef43ebe881c4dd69284295e23553a802fdb04c9fab94d"}
Jul 31 12:16:18.742061 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:18.742041    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kh24k" event={"ID":"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c","Type":"ContainerStarted","Data":"1370503c4cca05a7e3f22d16d846ce874572b7c81098c173565e718f176d3ec9"}
Jul 31 12:16:19.259337 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.259300    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:16:19.259859 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:16:19.259481    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 12:16:19.259859 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:16:19.259563    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls podName:6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa nodeName:}" failed. No retries permitted until 2026-07-31 12:16:51.259541259 +0000 UTC m=+99.246762779 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls") pod "dns-default-9d7l8" (UID: "6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa") : secret "dns-default-metrics-tls" not found
Jul 31 12:16:19.646927 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.646865    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"]
Jul 31 12:16:19.668906 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.668870    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"]
Jul 31 12:16:19.668906 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.668908    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"]
Jul 31 12:16:19.669807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.669723    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"
Jul 31 12:16:19.674261 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.674241    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 12:16:19.675184 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.675161    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cdqz9\""
Jul 31 12:16:19.675184 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.675175    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 12:16:19.676012 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.675481    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 31 12:16:19.676012 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.675833    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 12:16:19.685311 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.685243    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"]
Jul 31 12:16:19.685411 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.685367    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.688347 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.687777    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 31 12:16:19.688347 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.688151    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 31 12:16:19.688524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.688428    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 31 12:16:19.688664 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.688641    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 31 12:16:19.763186 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.763156    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-ca\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.763367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.763189    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-hub\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.763367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.763215    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2nbv\" (UniqueName: \"kubernetes.io/projected/9f3f96b0-5f89-4573-bcca-22a3362e88a1-kube-api-access-p2nbv\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.763367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.763261    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7bd00282-3740-4aef-8286-e6bfa0559ce1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74b8f9d77b-5svcx\" (UID: \"7bd00282-3740-4aef-8286-e6bfa0559ce1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"
Jul 31 12:16:19.763367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.763297    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwqrn\" (UniqueName: \"kubernetes.io/projected/7bd00282-3740-4aef-8286-e6bfa0559ce1-kube-api-access-jwqrn\") pod \"managed-serviceaccount-addon-agent-74b8f9d77b-5svcx\" (UID: \"7bd00282-3740-4aef-8286-e6bfa0559ce1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"
Jul 31 12:16:19.763553 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.763389    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3f96b0-5f89-4573-bcca-22a3362e88a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.763553 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.763435    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.763553 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.763539    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.864017 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.863982    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-ca\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.864017 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.864021    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-hub\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.864281 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.864044    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2nbv\" (UniqueName: \"kubernetes.io/projected/9f3f96b0-5f89-4573-bcca-22a3362e88a1-kube-api-access-p2nbv\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.864281 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.864085    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7bd00282-3740-4aef-8286-e6bfa0559ce1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74b8f9d77b-5svcx\" (UID: \"7bd00282-3740-4aef-8286-e6bfa0559ce1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"
Jul 31 12:16:19.864281 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.864108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwqrn\" (UniqueName: \"kubernetes.io/projected/7bd00282-3740-4aef-8286-e6bfa0559ce1-kube-api-access-jwqrn\") pod \"managed-serviceaccount-addon-agent-74b8f9d77b-5svcx\" (UID: \"7bd00282-3740-4aef-8286-e6bfa0559ce1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"
Jul 31 12:16:19.864281 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.864161    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3f96b0-5f89-4573-bcca-22a3362e88a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.864281 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.864198    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.864281 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.864229    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.865051 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.865024    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3f96b0-5f89-4573-bcca-22a3362e88a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.866929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.866888    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7bd00282-3740-4aef-8286-e6bfa0559ce1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74b8f9d77b-5svcx\" (UID: \"7bd00282-3740-4aef-8286-e6bfa0559ce1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"
Jul 31 12:16:19.867028 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.866975    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-ca\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.867220 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.867201    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.867376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.867354    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-hub\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.867534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.867514    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f3f96b0-5f89-4573-bcca-22a3362e88a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.871893 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.871863    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2nbv\" (UniqueName: \"kubernetes.io/projected/9f3f96b0-5f89-4573-bcca-22a3362e88a1-kube-api-access-p2nbv\") pod \"cluster-proxy-proxy-agent-654ff7bd8f-bhpw2\" (UID: \"9f3f96b0-5f89-4573-bcca-22a3362e88a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:19.872202 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.872183    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwqrn\" (UniqueName: \"kubernetes.io/projected/7bd00282-3740-4aef-8286-e6bfa0559ce1-kube-api-access-jwqrn\") pod \"managed-serviceaccount-addon-agent-74b8f9d77b-5svcx\" (UID: \"7bd00282-3740-4aef-8286-e6bfa0559ce1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"
Jul 31 12:16:19.991365 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:19.991287    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"
Jul 31 12:16:20.001511 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:20.001480    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:16:20.158908 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:20.158713    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx"]
Jul 31 12:16:20.162731 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:16:20.162697    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd00282_3740_4aef_8286_e6bfa0559ce1.slice/crio-84a2fbc12ad5ac69091d96af7d30e0fc5e1dbb54d3b3d67063ca2b1fe23ef870 WatchSource:0}: Error finding container 84a2fbc12ad5ac69091d96af7d30e0fc5e1dbb54d3b3d67063ca2b1fe23ef870: Status 404 returned error can't find the container with id 84a2fbc12ad5ac69091d96af7d30e0fc5e1dbb54d3b3d67063ca2b1fe23ef870
Jul 31 12:16:20.182822 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:20.182796    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"]
Jul 31 12:16:20.186455 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:16:20.186423    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f3f96b0_5f89_4573_bcca_22a3362e88a1.slice/crio-0893225574f38e08f7c334fc5831fabbd798fb073293667c33e391ee1d743c09 WatchSource:0}: Error finding container 0893225574f38e08f7c334fc5831fabbd798fb073293667c33e391ee1d743c09: Status 404 returned error can't find the container with id 0893225574f38e08f7c334fc5831fabbd798fb073293667c33e391ee1d743c09
Jul 31 12:16:20.749482 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:20.749446    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx" event={"ID":"7bd00282-3740-4aef-8286-e6bfa0559ce1","Type":"ContainerStarted","Data":"84a2fbc12ad5ac69091d96af7d30e0fc5e1dbb54d3b3d67063ca2b1fe23ef870"}
Jul 31 12:16:20.750719 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:20.750679    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" event={"ID":"9f3f96b0-5f89-4573-bcca-22a3362e88a1","Type":"ContainerStarted","Data":"0893225574f38e08f7c334fc5831fabbd798fb073293667c33e391ee1d743c09"}
Jul 31 12:16:27.778700 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.778668    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx" event={"ID":"7bd00282-3740-4aef-8286-e6bfa0559ce1","Type":"ContainerStarted","Data":"0844ead20b21dc05cb6c31b8f7ee81aacbcfa5ab402c00d121c52c7350a4b865"}
Jul 31 12:16:27.780539 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.780512    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" event={"ID":"9f3f96b0-5f89-4573-bcca-22a3362e88a1","Type":"ContainerStarted","Data":"4ae005658a71920d000c7392a6ee056ac3a31e53c5e1d2b02d13c7afe50b6351"}
Jul 31 12:16:27.780655 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.780546    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" event={"ID":"9f3f96b0-5f89-4573-bcca-22a3362e88a1","Type":"ContainerStarted","Data":"0ece64bb89ffffc286f6ce0ca8bf01d204d4b86b409786fce0ef632313cc8997"}
Jul 31 12:16:27.780655 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.780562    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" event={"ID":"9f3f96b0-5f89-4573-bcca-22a3362e88a1","Type":"ContainerStarted","Data":"b27d553170c3afd6fad841148888f162fceaf703a1298a0480edf9ec69ec44f5"}
Jul 31 12:16:27.781956 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.781937    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kh24k" event={"ID":"e729cbc6-e3c4-491f-9f6a-4b843eeb5b1c","Type":"ContainerStarted","Data":"e6bdf61be936e1c22d60943aeb429d30fa6810888547154b1a24d80851c02739"}
Jul 31 12:16:27.782061 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.782009    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:16:27.783310 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.783292    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pwtgz" event={"ID":"e0a25e8a-583a-4194-a8df-ff565cac7a43","Type":"ContainerStarted","Data":"7daad2ea2d70f111da1a475f4e5fd2f1334ac174a06a0c3a543c599d44fd2127"}
Jul 31 12:16:27.793912 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.793877    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx" podStartSLOduration=1.7991981940000001 podStartE2EDuration="8.793866076s" totalImagesPullingTime="6.994667882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:16:27.792848063 +0000 UTC m=+75.780069604" watchObservedRunningTime="2026-07-31 12:16:27.793866076 +0000 UTC m=+75.781087653"
Jul 31 12:16:27.809431 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.809393    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" podStartSLOduration=1.824493967 podStartE2EDuration="8.809380107s" totalImagesPullingTime="6.98488614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:16:27.808495016 +0000 UTC m=+75.795716557" watchObservedRunningTime="2026-07-31 12:16:27.809380107 +0000 UTC m=+75.796601652"
Jul 31 12:16:27.822528 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.822493    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kh24k" podStartSLOduration=67.284067136 podStartE2EDuration="1m15.822481403s" totalImagesPullingTime="8.538414267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:16:27.821724618 +0000 UTC m=+75.808946159" watchObservedRunningTime="2026-07-31 12:16:27.822481403 +0000 UTC m=+75.809702939"
Jul 31 12:16:27.834930 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:27.834892    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pwtgz" podStartSLOduration=67.250027375 podStartE2EDuration="1m15.834879837s" totalImagesPullingTime="8.584852462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:16:27.833995946 +0000 UTC m=+75.821217486" watchObservedRunningTime="2026-07-31 12:16:27.834879837 +0000 UTC m=+75.822101376"
Jul 31 12:16:51.280183 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:51.280150    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:16:51.280533 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:16:51.280285    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 12:16:51.280533 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:16:51.280340    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls podName:6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa nodeName:}" failed. No retries permitted until 2026-07-31 12:17:55.280326668 +0000 UTC m=+163.267548185 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls") pod "dns-default-9d7l8" (UID: "6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa") : secret "dns-default-metrics-tls" not found
Jul 31 12:16:58.788894 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:16:58.788865    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kh24k"
Jul 31 12:17:22.189690 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:17:22.189639    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:17:22.190172 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:17:22.189789    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 12:17:22.190172 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:17:22.189870    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:24.189852795 +0000 UTC m=+252.177074314 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : secret "metrics-daemon-secret" not found
Jul 31 12:17:50.382554 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:17:50.382503    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9d7l8" podUID="6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa"
Jul 31 12:17:50.497724 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:17:50.497683    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:17:51.021426 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:17:51.021399    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:17:55.317530 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:17:55.317474    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:17:55.317949 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:17:55.317619    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 12:17:55.317949 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:17:55.317680    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls podName:6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa nodeName:}" failed. No retries permitted until 2026-07-31 12:19:57.31766275 +0000 UTC m=+285.304884287 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls") pod "dns-default-9d7l8" (UID: "6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa") : secret "dns-default-metrics-tls" not found
Jul 31 12:18:05.479454 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:18:05.479359    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:18:28.120781 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:18:28.120742    2586 generic.go:356] "Generic (PLEG): container finished" podID="7bd00282-3740-4aef-8286-e6bfa0559ce1" containerID="0844ead20b21dc05cb6c31b8f7ee81aacbcfa5ab402c00d121c52c7350a4b865" exitCode=255
Jul 31 12:18:28.121291 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:18:28.120786    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx" event={"ID":"7bd00282-3740-4aef-8286-e6bfa0559ce1","Type":"ContainerDied","Data":"0844ead20b21dc05cb6c31b8f7ee81aacbcfa5ab402c00d121c52c7350a4b865"}
Jul 31 12:18:28.121291 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:18:28.121147    2586 scope.go:117] "RemoveContainer" containerID="0844ead20b21dc05cb6c31b8f7ee81aacbcfa5ab402c00d121c52c7350a4b865"
Jul 31 12:18:29.124823 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:18:29.124790    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b8f9d77b-5svcx" event={"ID":"7bd00282-3740-4aef-8286-e6bfa0559ce1","Type":"ContainerStarted","Data":"a4d5315a13e7c4164c69a188be3a25b55392db7ed9bbcd9cb1ca754cab7fb131"}
Jul 31 12:19:24.286042 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:24.286012    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:19:24.286538 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:24.286128    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 12:19:24.286538 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:24.286174    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs podName:7e2e533b-8f29-4a31-a152-c3f07d8d95f1 nodeName:}" failed. No retries permitted until 2026-07-31 12:21:26.286160803 +0000 UTC m=+374.273382320 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs") pod "network-metrics-daemon-cf94v" (UID: "7e2e533b-8f29-4a31-a152-c3f07d8d95f1") : secret "metrics-daemon-secret" not found
Jul 31 12:19:28.495027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.494998    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-74qwb"]
Jul 31 12:19:28.496887 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.496872    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.499337 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.499302    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 12:19:28.499452 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.499366    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 12:19:28.500276 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.500248    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qpgl9\""
Jul 31 12:19:28.500349 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.500285    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 12:19:28.500349 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.500309    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 12:19:28.508421 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.508401    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-74qwb"]
Jul 31 12:19:28.611814 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.611785    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b1b9f55-0c73-4d10-bb72-96adf76f8543-crio-socket\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.611814 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.611812    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.612005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.611833    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b1b9f55-0c73-4d10-bb72-96adf76f8543-data-volume\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.612005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.611851    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6sd7\" (UniqueName: \"kubernetes.io/projected/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-api-access-n6sd7\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.612005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.611888    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.713086 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.713058    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b1b9f55-0c73-4d10-bb72-96adf76f8543-data-volume\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.713252 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.713101    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6sd7\" (UniqueName: \"kubernetes.io/projected/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-api-access-n6sd7\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.713252 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.713130    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.713252 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.713218    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b1b9f55-0c73-4d10-bb72-96adf76f8543-crio-socket\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.713252 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.713243    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.713473 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:28.713307    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:28.713473 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:28.713340    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 12:19:28.713473 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.713351    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b1b9f55-0c73-4d10-bb72-96adf76f8543-crio-socket\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.713473 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:28.713374    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:29.213358625 +0000 UTC m=+257.200580143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:28.713473 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:28.713406    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:29.213389991 +0000 UTC m=+257.200611514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : configmap "kube-rbac-proxy" not found
Jul 31 12:19:28.713473 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.713432    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b1b9f55-0c73-4d10-bb72-96adf76f8543-data-volume\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:28.721511 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:28.721486    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6sd7\" (UniqueName: \"kubernetes.io/projected/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-api-access-n6sd7\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:29.217150 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:29.217115    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:29.217350 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:29.217161    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:29.217350 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:29.217236    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 12:19:29.217350 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:29.217251    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:29.217350 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:29.217328    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:30.217313981 +0000 UTC m=+258.204535499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:29.217350 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:29.217342    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:30.217335259 +0000 UTC m=+258.204556776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : configmap "kube-rbac-proxy" not found
Jul 31 12:19:30.224100 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:30.224053    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:30.224100 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:30.224111    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:30.224617 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:30.224180    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 12:19:30.224617 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:30.224210    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:30.224617 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:30.224253    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:32.224236991 +0000 UTC m=+260.211458516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:30.224617 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:30.224283    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:32.224260169 +0000 UTC m=+260.211481687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : configmap "kube-rbac-proxy" not found
Jul 31 12:19:30.648719 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:30.648690    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-plkjn_bfd73de9-7b12-4de5-ab9b-875c7bf9d62c/dns-node-resolver/0.log"
Jul 31 12:19:31.426724 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.426694    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-pqwwp"]
Jul 31 12:19:31.428725 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.428708    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.431192 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.431172    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 31 12:19:31.432167 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.432142    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 31 12:19:31.432167 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.432153    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 31 12:19:31.432328 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.432181    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 31 12:19:31.432328 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.432198    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8xqgq\""
Jul 31 12:19:31.437432 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.437413    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-pqwwp"]
Jul 31 12:19:31.533433 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.533402    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/09b96960-d446-452f-aa13-07ac562a1e38-signing-key\") pod \"service-ca-98c88fbb5-pqwwp\" (UID: \"09b96960-d446-452f-aa13-07ac562a1e38\") " pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.533582 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.533442    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nhbs\" (UniqueName: \"kubernetes.io/projected/09b96960-d446-452f-aa13-07ac562a1e38-kube-api-access-7nhbs\") pod \"service-ca-98c88fbb5-pqwwp\" (UID: \"09b96960-d446-452f-aa13-07ac562a1e38\") " pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.533582 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.533461    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/09b96960-d446-452f-aa13-07ac562a1e38-signing-cabundle\") pod \"service-ca-98c88fbb5-pqwwp\" (UID: \"09b96960-d446-452f-aa13-07ac562a1e38\") " pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.634683 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.634652    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/09b96960-d446-452f-aa13-07ac562a1e38-signing-key\") pod \"service-ca-98c88fbb5-pqwwp\" (UID: \"09b96960-d446-452f-aa13-07ac562a1e38\") " pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.634779 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.634691    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nhbs\" (UniqueName: \"kubernetes.io/projected/09b96960-d446-452f-aa13-07ac562a1e38-kube-api-access-7nhbs\") pod \"service-ca-98c88fbb5-pqwwp\" (UID: \"09b96960-d446-452f-aa13-07ac562a1e38\") " pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.634779 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.634709    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/09b96960-d446-452f-aa13-07ac562a1e38-signing-cabundle\") pod \"service-ca-98c88fbb5-pqwwp\" (UID: \"09b96960-d446-452f-aa13-07ac562a1e38\") " pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.635322 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.635296    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/09b96960-d446-452f-aa13-07ac562a1e38-signing-cabundle\") pod \"service-ca-98c88fbb5-pqwwp\" (UID: \"09b96960-d446-452f-aa13-07ac562a1e38\") " pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.637113 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.637095    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/09b96960-d446-452f-aa13-07ac562a1e38-signing-key\") pod \"service-ca-98c88fbb5-pqwwp\" (UID: \"09b96960-d446-452f-aa13-07ac562a1e38\") " pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.643252 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.643224    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nhbs\" (UniqueName: \"kubernetes.io/projected/09b96960-d446-452f-aa13-07ac562a1e38-kube-api-access-7nhbs\") pod \"service-ca-98c88fbb5-pqwwp\" (UID: \"09b96960-d446-452f-aa13-07ac562a1e38\") " pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.648716 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.648698    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jl8fm_70b1fc55-23f2-41da-981d-51d4f737c4e5/node-ca/0.log"
Jul 31 12:19:31.737653 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.737574    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp"
Jul 31 12:19:31.853017 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:31.852993    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-pqwwp"]
Jul 31 12:19:31.855799 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:19:31.855776    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b96960_d446_452f_aa13_07ac562a1e38.slice/crio-ff6a8f6c14c0fabc5008b8232bc7d547a1e666f7fd5b47e2dd6026eb2555e8e0 WatchSource:0}: Error finding container ff6a8f6c14c0fabc5008b8232bc7d547a1e666f7fd5b47e2dd6026eb2555e8e0: Status 404 returned error can't find the container with id ff6a8f6c14c0fabc5008b8232bc7d547a1e666f7fd5b47e2dd6026eb2555e8e0
Jul 31 12:19:32.049707 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:32.049622    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2qljl_7b7772db-0980-4ddb-ade7-cf9c5ec44a40/serve-healthcheck-canary/0.log"
Jul 31 12:19:32.240165 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:32.240133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:32.240377 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:32.240177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:32.240377 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:32.240292    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 12:19:32.240377 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:32.240367    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:36.240346747 +0000 UTC m=+264.227568265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : configmap "kube-rbac-proxy" not found
Jul 31 12:19:32.240559 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:32.240301    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:32.240559 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:32.240427    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:36.24041671 +0000 UTC m=+264.227638229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:32.301106 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:32.301021    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp" event={"ID":"09b96960-d446-452f-aa13-07ac562a1e38","Type":"ContainerStarted","Data":"ff6a8f6c14c0fabc5008b8232bc7d547a1e666f7fd5b47e2dd6026eb2555e8e0"}
Jul 31 12:19:34.309061 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:34.308983    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp" event={"ID":"09b96960-d446-452f-aa13-07ac562a1e38","Type":"ContainerStarted","Data":"bf8fe2f0bef7f4ea95969d160bc69ef1495ca9e7f85394329561bde83cabf316"}
Jul 31 12:19:34.334835 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:34.334790    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-pqwwp" podStartSLOduration=1.616826031 podStartE2EDuration="3.334774666s" totalImagesPullingTime="1.717948635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:19:34.33335433 +0000 UTC m=+262.320575873" watchObservedRunningTime="2026-07-31 12:19:34.334774666 +0000 UTC m=+262.321996206"
Jul 31 12:19:36.270972 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:36.270936    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:36.271450 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:36.271014    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:36.271450 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:36.271100    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 12:19:36.271450 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:36.271100    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:36.271450 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:36.271184    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:44.271168715 +0000 UTC m=+272.258390233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : configmap "kube-rbac-proxy" not found
Jul 31 12:19:36.271450 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:36.271245    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:19:44.27122768 +0000 UTC m=+272.258449204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:44.336660 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:44.336622    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:44.337058 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:44.336671    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:44.337058 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:44.336772    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:44.337058 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:44.336829    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls podName:6b1b9f55-0c73-4d10-bb72-96adf76f8543 nodeName:}" failed. No retries permitted until 2026-07-31 12:20:00.336811526 +0000 UTC m=+288.324033045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74qwb" (UID: "6b1b9f55-0c73-4d10-bb72-96adf76f8543") : secret "insights-runtime-extractor-tls" not found
Jul 31 12:19:44.337182 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:44.337136    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b1b9f55-0c73-4d10-bb72-96adf76f8543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:19:49.495413 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.495385    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-869447d669-jp8c2"]
Jul 31 12:19:49.497936 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.497919    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.500482 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.500456    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 12:19:49.500482 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.500470    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tjczm\""
Jul 31 12:19:49.500482 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.500461    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 12:19:49.501420 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.501402    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 12:19:49.505387 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.505368    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 12:19:49.509461 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.509443    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869447d669-jp8c2"]
Jul 31 12:19:49.575116 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.575084    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de572c0f-8fb3-4369-9abd-ecf748dfd05b-bound-sa-token\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.575298 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.575141    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de572c0f-8fb3-4369-9abd-ecf748dfd05b-installation-pull-secrets\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.575298 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.575196    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ssk\" (UniqueName: \"kubernetes.io/projected/de572c0f-8fb3-4369-9abd-ecf748dfd05b-kube-api-access-d4ssk\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.575298 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.575222    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de572c0f-8fb3-4369-9abd-ecf748dfd05b-image-registry-private-configuration\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.575298 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.575248    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de572c0f-8fb3-4369-9abd-ecf748dfd05b-ca-trust-extracted\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.575298 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.575279    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de572c0f-8fb3-4369-9abd-ecf748dfd05b-registry-tls\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.575489 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.575311    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de572c0f-8fb3-4369-9abd-ecf748dfd05b-registry-certificates\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.575489 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.575346    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de572c0f-8fb3-4369-9abd-ecf748dfd05b-trusted-ca\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.675761 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.675731    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de572c0f-8fb3-4369-9abd-ecf748dfd05b-trusted-ca\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.675915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.675774    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de572c0f-8fb3-4369-9abd-ecf748dfd05b-bound-sa-token\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.675915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.675818    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de572c0f-8fb3-4369-9abd-ecf748dfd05b-installation-pull-secrets\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.675915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.675850    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ssk\" (UniqueName: \"kubernetes.io/projected/de572c0f-8fb3-4369-9abd-ecf748dfd05b-kube-api-access-d4ssk\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.675915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.675887    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de572c0f-8fb3-4369-9abd-ecf748dfd05b-image-registry-private-configuration\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.676110 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.675917    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de572c0f-8fb3-4369-9abd-ecf748dfd05b-ca-trust-extracted\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.676110 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.675943    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de572c0f-8fb3-4369-9abd-ecf748dfd05b-registry-tls\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.676110 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.675985    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de572c0f-8fb3-4369-9abd-ecf748dfd05b-registry-certificates\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.676612 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.676582    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de572c0f-8fb3-4369-9abd-ecf748dfd05b-ca-trust-extracted\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.676785 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.676764    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de572c0f-8fb3-4369-9abd-ecf748dfd05b-registry-certificates\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.676865 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.676773    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de572c0f-8fb3-4369-9abd-ecf748dfd05b-trusted-ca\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.678311 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.678292    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de572c0f-8fb3-4369-9abd-ecf748dfd05b-image-registry-private-configuration\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.678809 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.678788    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de572c0f-8fb3-4369-9abd-ecf748dfd05b-registry-tls\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.678933 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.678912    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de572c0f-8fb3-4369-9abd-ecf748dfd05b-installation-pull-secrets\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.683203 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.683182    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de572c0f-8fb3-4369-9abd-ecf748dfd05b-bound-sa-token\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.683656 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.683577    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ssk\" (UniqueName: \"kubernetes.io/projected/de572c0f-8fb3-4369-9abd-ecf748dfd05b-kube-api-access-d4ssk\") pod \"image-registry-869447d669-jp8c2\" (UID: \"de572c0f-8fb3-4369-9abd-ecf748dfd05b\") " pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.807133 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.807058    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:49.929490 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:49.929397    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869447d669-jp8c2"]
Jul 31 12:19:49.931864 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:19:49.931836    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde572c0f_8fb3_4369_9abd_ecf748dfd05b.slice/crio-14e83f7265b518c71820bc3310b91e6f28f4afe5ae6b9a2858a7addbee1b04f2 WatchSource:0}: Error finding container 14e83f7265b518c71820bc3310b91e6f28f4afe5ae6b9a2858a7addbee1b04f2: Status 404 returned error can't find the container with id 14e83f7265b518c71820bc3310b91e6f28f4afe5ae6b9a2858a7addbee1b04f2
Jul 31 12:19:50.363495 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:50.363453    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869447d669-jp8c2" event={"ID":"de572c0f-8fb3-4369-9abd-ecf748dfd05b","Type":"ContainerStarted","Data":"6476a31deb7a0a5b16241bf4370f614b9ca33cf8f9e27ae791c5e7e690c63d96"}
Jul 31 12:19:50.363674 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:50.363530    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869447d669-jp8c2" event={"ID":"de572c0f-8fb3-4369-9abd-ecf748dfd05b","Type":"ContainerStarted","Data":"14e83f7265b518c71820bc3310b91e6f28f4afe5ae6b9a2858a7addbee1b04f2"}
Jul 31 12:19:50.363674 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:50.363568    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:19:50.383226 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:50.383182    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-869447d669-jp8c2" podStartSLOduration=1.383166779 podStartE2EDuration="1.383166779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:19:50.382346825 +0000 UTC m=+278.369568365" watchObservedRunningTime="2026-07-31 12:19:50.383166779 +0000 UTC m=+278.370388318"
Jul 31 12:19:54.022450 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:19:54.022409    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9d7l8" podUID="6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa"
Jul 31 12:19:54.374868 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:54.374790    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:19:57.331767 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:57.331731    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:19:57.334018 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:57.333996    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa-metrics-tls\") pod \"dns-default-9d7l8\" (UID: \"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa\") " pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:19:57.378327 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:57.378302    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ngjpv\""
Jul 31 12:19:57.386903 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:57.386886    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:19:57.504500 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:57.504473    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9d7l8"]
Jul 31 12:19:57.507457 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:19:57.507427    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b60bfd3_dea9_4b4a_aa2b_637e4c04ceaa.slice/crio-745e7a7cdba300f304fe0e60851630164dfa60616a9f3c0a048d1fb89906414d WatchSource:0}: Error finding container 745e7a7cdba300f304fe0e60851630164dfa60616a9f3c0a048d1fb89906414d: Status 404 returned error can't find the container with id 745e7a7cdba300f304fe0e60851630164dfa60616a9f3c0a048d1fb89906414d
Jul 31 12:19:58.389213 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:58.389175    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9d7l8" event={"ID":"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa","Type":"ContainerStarted","Data":"745e7a7cdba300f304fe0e60851630164dfa60616a9f3c0a048d1fb89906414d"}
Jul 31 12:19:59.393809 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:59.393727    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9d7l8" event={"ID":"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa","Type":"ContainerStarted","Data":"d680ff04f4e964ba2902100ce091d120cc1ea006f1b68f77a00f302335a042e8"}
Jul 31 12:19:59.393809 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:59.393776    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9d7l8" event={"ID":"6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa","Type":"ContainerStarted","Data":"fdfbd1eac5b72638b26d3117e5feb7d8eb742ea5f97a06d4f7559c11379c8776"}
Jul 31 12:19:59.394182 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:59.393860    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:19:59.411096 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:19:59.411055    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9d7l8" podStartSLOduration=250.846158353 podStartE2EDuration="4m12.411042009s" totalImagesPullingTime="1.564883656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:19:59.410561196 +0000 UTC m=+287.397782749" watchObservedRunningTime="2026-07-31 12:19:59.411042009 +0000 UTC m=+287.398263548"
Jul 31 12:20:00.354164 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:00.354131    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:20:00.356398 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:00.356376    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b1b9f55-0c73-4d10-bb72-96adf76f8543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74qwb\" (UID: \"6b1b9f55-0c73-4d10-bb72-96adf76f8543\") " pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:20:00.605175 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:00.605072    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-74qwb"
Jul 31 12:20:00.727303 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:00.727257    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-74qwb"]
Jul 31 12:20:00.730086 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:20:00.730054    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1b9f55_0c73_4d10_bb72_96adf76f8543.slice/crio-0ed09c99983a7a8c507bbde1834358298fbb4b63c73386f89d9491909ee17568 WatchSource:0}: Error finding container 0ed09c99983a7a8c507bbde1834358298fbb4b63c73386f89d9491909ee17568: Status 404 returned error can't find the container with id 0ed09c99983a7a8c507bbde1834358298fbb4b63c73386f89d9491909ee17568
Jul 31 12:20:01.402445 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:01.402405    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-74qwb" event={"ID":"6b1b9f55-0c73-4d10-bb72-96adf76f8543","Type":"ContainerStarted","Data":"012336de8a6f8ed589c463bec6a2f70618d0f15f220e3c5cedd7086f37aff25c"}
Jul 31 12:20:01.402611 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:01.402476    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-74qwb" event={"ID":"6b1b9f55-0c73-4d10-bb72-96adf76f8543","Type":"ContainerStarted","Data":"0ed09c99983a7a8c507bbde1834358298fbb4b63c73386f89d9491909ee17568"}
Jul 31 12:20:02.407365 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:02.407333    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-74qwb" event={"ID":"6b1b9f55-0c73-4d10-bb72-96adf76f8543","Type":"ContainerStarted","Data":"f57a9310ad6b64bc4c7d30bc77148cc287cbdd56d309375ca9022ba792bb42ce"}
Jul 31 12:20:03.412076 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:03.412044    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-74qwb" event={"ID":"6b1b9f55-0c73-4d10-bb72-96adf76f8543","Type":"ContainerStarted","Data":"df698a4534b0b5f9950959c8df04c3f1d4c0d9e9bc29161d818e5e47441959be"}
Jul 31 12:20:03.429509 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:03.429465    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-74qwb" podStartSLOduration=33.228902184 podStartE2EDuration="35.429449656s" totalImagesPullingTime="2.200547472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 12:19:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:20:03.428329775 +0000 UTC m=+291.415551315" watchObservedRunningTime="2026-07-31 12:20:03.429449656 +0000 UTC m=+291.416671196"
Jul 31 12:20:07.243679 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.243642    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c86rz"]
Jul 31 12:20:07.246012 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.245990    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.248591 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.248567    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c5xjv\""
Jul 31 12:20:07.248718 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.248569    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 12:20:07.248718 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.248610    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 12:20:07.249459 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.249438    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 12:20:07.249602 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.249556    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 12:20:07.249673 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.249606    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 12:20:07.249673 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.249648    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 12:20:07.302204 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.302174    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.302497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.302213    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.302497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.302245    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05ee5666-45cf-495d-8c43-5f4435d6f7d9-sys\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.302497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.302348    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05ee5666-45cf-495d-8c43-5f4435d6f7d9-metrics-client-ca\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.302497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.302386    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g95f9\" (UniqueName: \"kubernetes.io/projected/05ee5666-45cf-495d-8c43-5f4435d6f7d9-kube-api-access-g95f9\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.302497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.302434    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-tls\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.302497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.302458    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-wtmp\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.302497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.302476    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/05ee5666-45cf-495d-8c43-5f4435d6f7d9-root\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.302497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.302489    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-textfile\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403499 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403457    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-wtmp\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403499 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403505    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/05ee5666-45cf-495d-8c43-5f4435d6f7d9-root\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403715 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403569    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/05ee5666-45cf-495d-8c43-5f4435d6f7d9-root\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403715 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403618    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-textfile\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403715 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403626    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-wtmp\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403715 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403665    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403715 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403698    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403958 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403728    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05ee5666-45cf-495d-8c43-5f4435d6f7d9-sys\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403958 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403797    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05ee5666-45cf-495d-8c43-5f4435d6f7d9-metrics-client-ca\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403958 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403807    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05ee5666-45cf-495d-8c43-5f4435d6f7d9-sys\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403958 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403855    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g95f9\" (UniqueName: \"kubernetes.io/projected/05ee5666-45cf-495d-8c43-5f4435d6f7d9-kube-api-access-g95f9\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.403958 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403920    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-tls\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.404195 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.403989    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-textfile\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.404323 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.404296    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.404636 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.404617    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05ee5666-45cf-495d-8c43-5f4435d6f7d9-metrics-client-ca\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.406040 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.406020    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.406415 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.406401    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/05ee5666-45cf-495d-8c43-5f4435d6f7d9-node-exporter-tls\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.411575 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.411556    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g95f9\" (UniqueName: \"kubernetes.io/projected/05ee5666-45cf-495d-8c43-5f4435d6f7d9-kube-api-access-g95f9\") pod \"node-exporter-c86rz\" (UID: \"05ee5666-45cf-495d-8c43-5f4435d6f7d9\") " pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.556158 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:07.556084    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c86rz"
Jul 31 12:20:07.565758 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:20:07.565732    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ee5666_45cf_495d_8c43_5f4435d6f7d9.slice/crio-c32f47a0b50ea7df841fb9ac7932fbf9cf504fd88b8fdf422fb36623ca367be1 WatchSource:0}: Error finding container c32f47a0b50ea7df841fb9ac7932fbf9cf504fd88b8fdf422fb36623ca367be1: Status 404 returned error can't find the container with id c32f47a0b50ea7df841fb9ac7932fbf9cf504fd88b8fdf422fb36623ca367be1
Jul 31 12:20:08.429317 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:08.429281    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c86rz" event={"ID":"05ee5666-45cf-495d-8c43-5f4435d6f7d9","Type":"ContainerStarted","Data":"c32f47a0b50ea7df841fb9ac7932fbf9cf504fd88b8fdf422fb36623ca367be1"}
Jul 31 12:20:08.480548 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:20:08.480513    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cf94v" podUID="7e2e533b-8f29-4a31-a152-c3f07d8d95f1"
Jul 31 12:20:09.399967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:09.399941    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9d7l8"
Jul 31 12:20:09.435251 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:09.435219    2586 generic.go:356] "Generic (PLEG): container finished" podID="05ee5666-45cf-495d-8c43-5f4435d6f7d9" containerID="d0ec27bc7807efb3ebaa4ce5f6d4bff5c5f3edc7812ea17072117964d81a8ceb" exitCode=0
Jul 31 12:20:09.435724 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:09.435290    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c86rz" event={"ID":"05ee5666-45cf-495d-8c43-5f4435d6f7d9","Type":"ContainerDied","Data":"d0ec27bc7807efb3ebaa4ce5f6d4bff5c5f3edc7812ea17072117964d81a8ceb"}
Jul 31 12:20:09.811651 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:09.811573    2586 patch_prober.go:28] interesting pod/image-registry-869447d669-jp8c2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 31 12:20:09.811651 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:09.811625    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-869447d669-jp8c2" podUID="de572c0f-8fb3-4369-9abd-ecf748dfd05b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 12:20:10.440544 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:10.440511    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c86rz" event={"ID":"05ee5666-45cf-495d-8c43-5f4435d6f7d9","Type":"ContainerStarted","Data":"11f5c784255174fd7218eb4a113c82a6b5623b64bd6ba1e8d379136927e9dfc1"}
Jul 31 12:20:10.440931 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:10.440567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c86rz" event={"ID":"05ee5666-45cf-495d-8c43-5f4435d6f7d9","Type":"ContainerStarted","Data":"a4968e304113f6bcbf2b47fe3ba8b8895582af2db40551b7a2cbe5e08cfcaf75"}
Jul 31 12:20:10.461203 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:10.461150    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c86rz" podStartSLOduration=2.275831766 podStartE2EDuration="3.461134627s" totalImagesPullingTime="1.124251558s" totalInitContainerRuntime="61.051303ms" isStatefulPod=true podCreationTimestamp="2026-07-31 12:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:20:10.460727208 +0000 UTC m=+298.447948747" watchObservedRunningTime="2026-07-31 12:20:10.461134627 +0000 UTC m=+298.448356166"
Jul 31 12:20:11.370468 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:11.370440    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-869447d669-jp8c2"
Jul 31 12:20:12.372800 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:12.372773    2586 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 12:20:20.002683 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:20.002643    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" podUID="9f3f96b0-5f89-4573-bcca-22a3362e88a1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 31 12:20:20.479360 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:20.479332    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:20:30.002689 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:30.002647    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" podUID="9f3f96b0-5f89-4573-bcca-22a3362e88a1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 31 12:20:40.002761 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:40.002724    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" podUID="9f3f96b0-5f89-4573-bcca-22a3362e88a1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 31 12:20:40.003154 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:40.002787    2586 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2"
Jul 31 12:20:40.003258 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:40.003229    2586 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"4ae005658a71920d000c7392a6ee056ac3a31e53c5e1d2b02d13c7afe50b6351"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 31 12:20:40.003317 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:40.003294    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" podUID="9f3f96b0-5f89-4573-bcca-22a3362e88a1" containerName="service-proxy" containerID="cri-o://4ae005658a71920d000c7392a6ee056ac3a31e53c5e1d2b02d13c7afe50b6351" gracePeriod=30
Jul 31 12:20:40.533416 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:40.533381    2586 generic.go:356] "Generic (PLEG): container finished" podID="9f3f96b0-5f89-4573-bcca-22a3362e88a1" containerID="4ae005658a71920d000c7392a6ee056ac3a31e53c5e1d2b02d13c7afe50b6351" exitCode=2
Jul 31 12:20:40.533571 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:40.533454    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" event={"ID":"9f3f96b0-5f89-4573-bcca-22a3362e88a1","Type":"ContainerDied","Data":"4ae005658a71920d000c7392a6ee056ac3a31e53c5e1d2b02d13c7afe50b6351"}
Jul 31 12:20:40.533571 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:20:40.533496    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-654ff7bd8f-bhpw2" event={"ID":"9f3f96b0-5f89-4573-bcca-22a3362e88a1","Type":"ContainerStarted","Data":"2341242f192c686b385e39e1134d6b6807e6249dc4d0868f95a88614c42c15d0"}
Jul 31 12:21:26.384077 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:26.384032    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:21:26.386328 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:26.386307    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2e533b-8f29-4a31-a152-c3f07d8d95f1-metrics-certs\") pod \"network-metrics-daemon-cf94v\" (UID: \"7e2e533b-8f29-4a31-a152-c3f07d8d95f1\") " pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:21:26.482183 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:26.482157    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qqfkp\""
Jul 31 12:21:26.490021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:26.489993    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf94v"
Jul 31 12:21:26.610468 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:26.610439    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cf94v"]
Jul 31 12:21:26.613309 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:21:26.613259    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2e533b_8f29_4a31_a152_c3f07d8d95f1.slice/crio-485fd1e23b7dd6f4be68a21e796ca5550653bbf0ddcb9f9388b1fe1d3b77ae03 WatchSource:0}: Error finding container 485fd1e23b7dd6f4be68a21e796ca5550653bbf0ddcb9f9388b1fe1d3b77ae03: Status 404 returned error can't find the container with id 485fd1e23b7dd6f4be68a21e796ca5550653bbf0ddcb9f9388b1fe1d3b77ae03
Jul 31 12:21:26.615299 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:26.615282    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 12:21:26.675168 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:26.675143    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cf94v" event={"ID":"7e2e533b-8f29-4a31-a152-c3f07d8d95f1","Type":"ContainerStarted","Data":"485fd1e23b7dd6f4be68a21e796ca5550653bbf0ddcb9f9388b1fe1d3b77ae03"}
Jul 31 12:21:28.685714 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:28.685677    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cf94v" event={"ID":"7e2e533b-8f29-4a31-a152-c3f07d8d95f1","Type":"ContainerStarted","Data":"67804aafd45cdffd1a0a0beb3587815b78cfb97b2bdf114ca00686fc1c8a5c0b"}
Jul 31 12:21:28.686098 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:28.685731    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cf94v" event={"ID":"7e2e533b-8f29-4a31-a152-c3f07d8d95f1","Type":"ContainerStarted","Data":"a60554455053efdcec80ddc73837a9d6571148130d05b221e842d360aedfd11e"}
Jul 31 12:21:28.701277 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:21:28.701230    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cf94v" podStartSLOduration=375.607651657 podStartE2EDuration="6m16.701215786s" totalImagesPullingTime="1.093564129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:21:28.70057964 +0000 UTC m=+376.687801181" watchObservedRunningTime="2026-07-31 12:21:28.701215786 +0000 UTC m=+376.688437326"
Jul 31 12:24:09.723298 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.723202    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"]
Jul 31 12:24:09.725367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.725350    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.727829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.727806    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 31 12:24:09.727946 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.727837    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 31 12:24:09.727946 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.727932    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4n6bj\""
Jul 31 12:24:09.728323 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.728307    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 31 12:24:09.728392 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.728377    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 31 12:24:09.740319 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.740299    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"]
Jul 31 12:24:09.887502 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.887470    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28r7q\" (UniqueName: \"kubernetes.io/projected/28d114e9-c01c-4f60-8c23-b04aafdd2d38-kube-api-access-28r7q\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.887654 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.887517    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28d114e9-c01c-4f60-8c23-b04aafdd2d38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.887654 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.887589    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28d114e9-c01c-4f60-8c23-b04aafdd2d38-webhook-cert\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.887654 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.887620    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28d114e9-c01c-4f60-8c23-b04aafdd2d38-tmp\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.887654 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.887638    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28d114e9-c01c-4f60-8c23-b04aafdd2d38-manifests\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.988937 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.988860    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28d114e9-c01c-4f60-8c23-b04aafdd2d38-webhook-cert\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.988937 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.988891    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28d114e9-c01c-4f60-8c23-b04aafdd2d38-tmp\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.988937 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.988910    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28d114e9-c01c-4f60-8c23-b04aafdd2d38-manifests\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.989183 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.988956    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28r7q\" (UniqueName: \"kubernetes.io/projected/28d114e9-c01c-4f60-8c23-b04aafdd2d38-kube-api-access-28r7q\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.989183 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.989004    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28d114e9-c01c-4f60-8c23-b04aafdd2d38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.989437 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.989413    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28d114e9-c01c-4f60-8c23-b04aafdd2d38-manifests\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.991164 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.991144    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28d114e9-c01c-4f60-8c23-b04aafdd2d38-tmp\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.991251 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.991201    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28d114e9-c01c-4f60-8c23-b04aafdd2d38-webhook-cert\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:09.991407 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:09.991391    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28d114e9-c01c-4f60-8c23-b04aafdd2d38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:10.001536 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:10.001517    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28r7q\" (UniqueName: \"kubernetes.io/projected/28d114e9-c01c-4f60-8c23-b04aafdd2d38-kube-api-access-28r7q\") pod \"opendatahub-operator-controller-manager-5996b4548f-8pcwp\" (UID: \"28d114e9-c01c-4f60-8c23-b04aafdd2d38\") " pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:10.035182 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:10.035161    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:10.160049 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:10.160027    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"]
Jul 31 12:24:10.162861 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:24:10.162838    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d114e9_c01c_4f60_8c23_b04aafdd2d38.slice/crio-981309e2e409fb03c55b645e0a600f72004564ca06c1fae19a9dd6d9753aaaa1 WatchSource:0}: Error finding container 981309e2e409fb03c55b645e0a600f72004564ca06c1fae19a9dd6d9753aaaa1: Status 404 returned error can't find the container with id 981309e2e409fb03c55b645e0a600f72004564ca06c1fae19a9dd6d9753aaaa1
Jul 31 12:24:10.199870 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:10.199847    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp" event={"ID":"28d114e9-c01c-4f60-8c23-b04aafdd2d38","Type":"ContainerStarted","Data":"981309e2e409fb03c55b645e0a600f72004564ca06c1fae19a9dd6d9753aaaa1"}
Jul 31 12:24:13.212534 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:13.212492    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp" event={"ID":"28d114e9-c01c-4f60-8c23-b04aafdd2d38","Type":"ContainerStarted","Data":"ff427327186a6a09154a5ec7933d2b8fc427abfde676b876a63fbf4488e65804"}
Jul 31 12:24:14.217128 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:14.217097    2586 generic.go:356] "Generic (PLEG): container finished" podID="28d114e9-c01c-4f60-8c23-b04aafdd2d38" containerID="ff427327186a6a09154a5ec7933d2b8fc427abfde676b876a63fbf4488e65804" exitCode=0
Jul 31 12:24:14.217486 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:14.217160    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp" event={"ID":"28d114e9-c01c-4f60-8c23-b04aafdd2d38","Type":"ContainerDied","Data":"ff427327186a6a09154a5ec7933d2b8fc427abfde676b876a63fbf4488e65804"}
Jul 31 12:24:15.222661 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:15.222623    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp" event={"ID":"28d114e9-c01c-4f60-8c23-b04aafdd2d38","Type":"ContainerStarted","Data":"4d986c2f2ee4d3316795758426dcee640e387c17fe3071d236b8b1059dbcbd20"}
Jul 31 12:24:15.223054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:15.222733    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:15.243060 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:15.243017    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp" podStartSLOduration=2.015860649 podStartE2EDuration="6.242998409s" totalImagesPullingTime="3.162927733s" totalInitContainerRuntime="1.064210027s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:24:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:24:15.241494486 +0000 UTC m=+543.228716026" watchObservedRunningTime="2026-07-31 12:24:15.242998409 +0000 UTC m=+543.230219949"
Jul 31 12:24:26.228356 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:26.228321    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5996b4548f-8pcwp"
Jul 31 12:24:29.344904 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.344867    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"]
Jul 31 12:24:29.354475 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.354452    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.357489 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.357462    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 12:24:29.357489 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.357471    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pcrdc\""
Jul 31 12:24:29.358314 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.358300    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 31 12:24:29.358363 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.358329    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 12:24:29.359353 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.359341    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 31 12:24:29.372213 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.372189    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"]
Jul 31 12:24:29.373744 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.373719    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v6mcf"]
Jul 31 12:24:29.376145 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.376123    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:29.379416 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.379400    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 31 12:24:29.379731 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.379718    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-r5rkv\""
Jul 31 12:24:29.393452 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.391736    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v6mcf"]
Jul 31 12:24:29.421924 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.421893    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5656s\" (UniqueName: \"kubernetes.io/projected/de430a35-bac6-4020-a36c-8395a5c6e5d5-kube-api-access-5656s\") pod \"kube-auth-proxy-84cd848656-kxhxj\" (UID: \"de430a35-bac6-4020-a36c-8395a5c6e5d5\") " pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.422053 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.421931    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de430a35-bac6-4020-a36c-8395a5c6e5d5-tls-certs\") pod \"kube-auth-proxy-84cd848656-kxhxj\" (UID: \"de430a35-bac6-4020-a36c-8395a5c6e5d5\") " pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.422053 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.422020    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwkmj\" (UniqueName: \"kubernetes.io/projected/bdc9f096-64da-49ad-928f-c3abdcc63294-kube-api-access-rwkmj\") pod \"odh-model-controller-7998c67dc4-v6mcf\" (UID: \"bdc9f096-64da-49ad-928f-c3abdcc63294\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:29.422158 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.422061    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc9f096-64da-49ad-928f-c3abdcc63294-cert\") pod \"odh-model-controller-7998c67dc4-v6mcf\" (UID: \"bdc9f096-64da-49ad-928f-c3abdcc63294\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:29.422158 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.422083    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de430a35-bac6-4020-a36c-8395a5c6e5d5-tmp\") pod \"kube-auth-proxy-84cd848656-kxhxj\" (UID: \"de430a35-bac6-4020-a36c-8395a5c6e5d5\") " pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.523317 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.523283    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwkmj\" (UniqueName: \"kubernetes.io/projected/bdc9f096-64da-49ad-928f-c3abdcc63294-kube-api-access-rwkmj\") pod \"odh-model-controller-7998c67dc4-v6mcf\" (UID: \"bdc9f096-64da-49ad-928f-c3abdcc63294\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:29.523317 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.523321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc9f096-64da-49ad-928f-c3abdcc63294-cert\") pod \"odh-model-controller-7998c67dc4-v6mcf\" (UID: \"bdc9f096-64da-49ad-928f-c3abdcc63294\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:29.523563 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.523337    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de430a35-bac6-4020-a36c-8395a5c6e5d5-tmp\") pod \"kube-auth-proxy-84cd848656-kxhxj\" (UID: \"de430a35-bac6-4020-a36c-8395a5c6e5d5\") " pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.523563 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.523381    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5656s\" (UniqueName: \"kubernetes.io/projected/de430a35-bac6-4020-a36c-8395a5c6e5d5-kube-api-access-5656s\") pod \"kube-auth-proxy-84cd848656-kxhxj\" (UID: \"de430a35-bac6-4020-a36c-8395a5c6e5d5\") " pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.523563 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.523399    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de430a35-bac6-4020-a36c-8395a5c6e5d5-tls-certs\") pod \"kube-auth-proxy-84cd848656-kxhxj\" (UID: \"de430a35-bac6-4020-a36c-8395a5c6e5d5\") " pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.523563 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:24:29.523442    2586 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 31 12:24:29.523563 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:24:29.523533    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdc9f096-64da-49ad-928f-c3abdcc63294-cert podName:bdc9f096-64da-49ad-928f-c3abdcc63294 nodeName:}" failed. No retries permitted until 2026-07-31 12:24:30.023510814 +0000 UTC m=+558.010732336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bdc9f096-64da-49ad-928f-c3abdcc63294-cert") pod "odh-model-controller-7998c67dc4-v6mcf" (UID: "bdc9f096-64da-49ad-928f-c3abdcc63294") : secret "odh-model-controller-webhook-cert" not found
Jul 31 12:24:29.525793 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.525763    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de430a35-bac6-4020-a36c-8395a5c6e5d5-tmp\") pod \"kube-auth-proxy-84cd848656-kxhxj\" (UID: \"de430a35-bac6-4020-a36c-8395a5c6e5d5\") " pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.525910 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.525850    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de430a35-bac6-4020-a36c-8395a5c6e5d5-tls-certs\") pod \"kube-auth-proxy-84cd848656-kxhxj\" (UID: \"de430a35-bac6-4020-a36c-8395a5c6e5d5\") " pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.532075 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.532054    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5656s\" (UniqueName: \"kubernetes.io/projected/de430a35-bac6-4020-a36c-8395a5c6e5d5-kube-api-access-5656s\") pod \"kube-auth-proxy-84cd848656-kxhxj\" (UID: \"de430a35-bac6-4020-a36c-8395a5c6e5d5\") " pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.532296 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.532259    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwkmj\" (UniqueName: \"kubernetes.io/projected/bdc9f096-64da-49ad-928f-c3abdcc63294-kube-api-access-rwkmj\") pod \"odh-model-controller-7998c67dc4-v6mcf\" (UID: \"bdc9f096-64da-49ad-928f-c3abdcc63294\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:29.664940 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.664914    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"
Jul 31 12:24:29.826387 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:29.826360    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84cd848656-kxhxj"]
Jul 31 12:24:29.829407 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:24:29.829379    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde430a35_bac6_4020_a36c_8395a5c6e5d5.slice/crio-bd535984fe07b777c620fc8c90cfcbbb20010e1dadc35faf7c55ed69963e3aab WatchSource:0}: Error finding container bd535984fe07b777c620fc8c90cfcbbb20010e1dadc35faf7c55ed69963e3aab: Status 404 returned error can't find the container with id bd535984fe07b777c620fc8c90cfcbbb20010e1dadc35faf7c55ed69963e3aab
Jul 31 12:24:30.027022 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.026922    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc9f096-64da-49ad-928f-c3abdcc63294-cert\") pod \"odh-model-controller-7998c67dc4-v6mcf\" (UID: \"bdc9f096-64da-49ad-928f-c3abdcc63294\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:30.030052 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.030023    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc9f096-64da-49ad-928f-c3abdcc63294-cert\") pod \"odh-model-controller-7998c67dc4-v6mcf\" (UID: \"bdc9f096-64da-49ad-928f-c3abdcc63294\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:30.157823 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.157785    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9sgdg"]
Jul 31 12:24:30.160576 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.160560    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:24:30.163337 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.163304    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 31 12:24:30.163443 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.163425    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-sv654\""
Jul 31 12:24:30.173372 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.173343    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9sgdg"]
Jul 31 12:24:30.228683 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.228651    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8149d5f-9913-492e-a13b-10e4b66e4145-tls-certs\") pod \"model-serving-api-54d89bd5fc-9sgdg\" (UID: \"f8149d5f-9913-492e-a13b-10e4b66e4145\") " pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:24:30.228830 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.228693    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdgb\" (UniqueName: \"kubernetes.io/projected/f8149d5f-9913-492e-a13b-10e4b66e4145-kube-api-access-hvdgb\") pod \"model-serving-api-54d89bd5fc-9sgdg\" (UID: \"f8149d5f-9913-492e-a13b-10e4b66e4145\") " pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:24:30.280223 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.280142    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj" event={"ID":"de430a35-bac6-4020-a36c-8395a5c6e5d5","Type":"ContainerStarted","Data":"bd535984fe07b777c620fc8c90cfcbbb20010e1dadc35faf7c55ed69963e3aab"}
Jul 31 12:24:30.289797 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.289773    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:30.330048 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.330008    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdgb\" (UniqueName: \"kubernetes.io/projected/f8149d5f-9913-492e-a13b-10e4b66e4145-kube-api-access-hvdgb\") pod \"model-serving-api-54d89bd5fc-9sgdg\" (UID: \"f8149d5f-9913-492e-a13b-10e4b66e4145\") " pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:24:30.330236 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.330121    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8149d5f-9913-492e-a13b-10e4b66e4145-tls-certs\") pod \"model-serving-api-54d89bd5fc-9sgdg\" (UID: \"f8149d5f-9913-492e-a13b-10e4b66e4145\") " pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:24:30.332996 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.332842    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8149d5f-9913-492e-a13b-10e4b66e4145-tls-certs\") pod \"model-serving-api-54d89bd5fc-9sgdg\" (UID: \"f8149d5f-9913-492e-a13b-10e4b66e4145\") " pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:24:30.346853 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.346828    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdgb\" (UniqueName: \"kubernetes.io/projected/f8149d5f-9913-492e-a13b-10e4b66e4145-kube-api-access-hvdgb\") pod \"model-serving-api-54d89bd5fc-9sgdg\" (UID: \"f8149d5f-9913-492e-a13b-10e4b66e4145\") " pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:24:30.439138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.439110    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v6mcf"]
Jul 31 12:24:30.440958 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:24:30.440914    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc9f096_64da_49ad_928f_c3abdcc63294.slice/crio-4a18b73e6ed556689522d2739d7bad425fc80df1aa564250695d5a2d9b6ab591 WatchSource:0}: Error finding container 4a18b73e6ed556689522d2739d7bad425fc80df1aa564250695d5a2d9b6ab591: Status 404 returned error can't find the container with id 4a18b73e6ed556689522d2739d7bad425fc80df1aa564250695d5a2d9b6ab591
Jul 31 12:24:30.471500 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.471478    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:24:30.475125 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.475105    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"]
Jul 31 12:24:30.477106 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.477087    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.479547 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.479526    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 31 12:24:30.479647 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.479552    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 31 12:24:30.480141 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.480119    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 31 12:24:30.480298 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.480261    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rj97x\""
Jul 31 12:24:30.480376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.480359    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 12:24:30.492660 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.492602    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"]
Jul 31 12:24:30.532458 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.532399    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ec1b9-b258-468a-933f-a12feb4acd55-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.532458 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.532437    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.532627 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.532503    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.532627 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.532529    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.532627 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.532560    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.532627 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.532586    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtmcn\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-kube-api-access-mtmcn\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.532840 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.532645    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.599631 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.599600    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9sgdg"]
Jul 31 12:24:30.602898 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:24:30.602871    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8149d5f_9913_492e_a13b_10e4b66e4145.slice/crio-d2505013d49030af4353be360ef4ae1ca1c0cb6e1d448271ad3b52c1f0e1527b WatchSource:0}: Error finding container d2505013d49030af4353be360ef4ae1ca1c0cb6e1d448271ad3b52c1f0e1527b: Status 404 returned error can't find the container with id d2505013d49030af4353be360ef4ae1ca1c0cb6e1d448271ad3b52c1f0e1527b
Jul 31 12:24:30.633176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.633134    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.633406 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.633189    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ec1b9-b258-468a-933f-a12feb4acd55-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.633406 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.633225    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.633406 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.633306    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.633406 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.633345    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.633406 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.633386    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.633708 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.633429    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtmcn\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-kube-api-access-mtmcn\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.633817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.633727    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.636111 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.636064    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ec1b9-b258-468a-933f-a12feb4acd55-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.636225 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.636161    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.637054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.636412    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.637054 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.636424    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.641733 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.641711    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtmcn\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-kube-api-access-mtmcn\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.641926 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.641819    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-pgs78\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.788761 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.788680    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:30.947021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:30.946998    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"]
Jul 31 12:24:30.949957 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:24:30.949931    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54ec1b9_b258_468a_933f_a12feb4acd55.slice/crio-f3d32d9ddc371a028742561d0113f3f5ad440b1f9f83ccb4480a3f5f87c5c172 WatchSource:0}: Error finding container f3d32d9ddc371a028742561d0113f3f5ad440b1f9f83ccb4480a3f5f87c5c172: Status 404 returned error can't find the container with id f3d32d9ddc371a028742561d0113f3f5ad440b1f9f83ccb4480a3f5f87c5c172
Jul 31 12:24:31.286586 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:31.286477    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg" event={"ID":"f8149d5f-9913-492e-a13b-10e4b66e4145","Type":"ContainerStarted","Data":"d2505013d49030af4353be360ef4ae1ca1c0cb6e1d448271ad3b52c1f0e1527b"}
Jul 31 12:24:31.287972 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:31.287934    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf" event={"ID":"bdc9f096-64da-49ad-928f-c3abdcc63294","Type":"ContainerStarted","Data":"4a18b73e6ed556689522d2739d7bad425fc80df1aa564250695d5a2d9b6ab591"}
Jul 31 12:24:31.289246 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:31.289219    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78" event={"ID":"d54ec1b9-b258-468a-933f-a12feb4acd55","Type":"ContainerStarted","Data":"f3d32d9ddc371a028742561d0113f3f5ad440b1f9f83ccb4480a3f5f87c5c172"}
Jul 31 12:24:35.703582 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:35.703537    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:24:35.703966 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:35.703614    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:24:36.316113 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.316069    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj" event={"ID":"de430a35-bac6-4020-a36c-8395a5c6e5d5","Type":"ContainerStarted","Data":"5851828f78a501b6e8bb7012fd43c994ca4c2ea5d2ba0b9331f94f25cef66ce0"}
Jul 31 12:24:36.317842 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.317808    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg" event={"ID":"f8149d5f-9913-492e-a13b-10e4b66e4145","Type":"ContainerStarted","Data":"7f357a554f8df7de82afb92cb153f6fc70718b5ffdebf1755d635e8172e0efe7"}
Jul 31 12:24:36.317993 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.317927    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:24:36.319645 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.319615    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf" event={"ID":"bdc9f096-64da-49ad-928f-c3abdcc63294","Type":"ContainerStarted","Data":"ad1994f74d3a7bb0536e5f6097bd23114689a8b6ed1bbf5d53c1e6019e59c265"}
Jul 31 12:24:36.319779 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.319675    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:36.321480 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.321457    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78" event={"ID":"d54ec1b9-b258-468a-933f-a12feb4acd55","Type":"ContainerStarted","Data":"cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b"}
Jul 31 12:24:36.321675 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.321660    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:36.323583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.323559    2586 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-pgs78 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 12:24:36.323700 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.323609    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78" podUID="d54ec1b9-b258-468a-933f-a12feb4acd55" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 12:24:36.355291 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.355075    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84cd848656-kxhxj" podStartSLOduration=1.496363486 podStartE2EDuration="7.355053702s" totalImagesPullingTime="5.858690216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:24:36.335134013 +0000 UTC m=+564.322355567" watchObservedRunningTime="2026-07-31 12:24:36.355053702 +0000 UTC m=+564.342275243"
Jul 31 12:24:36.356443 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.356334    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf" podStartSLOduration=2.095252647 podStartE2EDuration="7.356320727s" totalImagesPullingTime="5.26106808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:24:36.354529541 +0000 UTC m=+564.341751083" watchObservedRunningTime="2026-07-31 12:24:36.356320727 +0000 UTC m=+564.343542269"
Jul 31 12:24:36.370980 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.370934    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg" podStartSLOduration=1.285602398 podStartE2EDuration="6.370914762s" totalImagesPullingTime="5.085312364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:24:36.369682822 +0000 UTC m=+564.356904366" watchObservedRunningTime="2026-07-31 12:24:36.370914762 +0000 UTC m=+564.358136303"
Jul 31 12:24:36.388281 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:36.388210    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78" podStartSLOduration=1.636875341 podStartE2EDuration="6.388191689s" totalImagesPullingTime="4.751316348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:24:36.387169732 +0000 UTC m=+564.374391277" watchObservedRunningTime="2026-07-31 12:24:36.388191689 +0000 UTC m=+564.375413231"
Jul 31 12:24:37.325841 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:37.325816    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:40.712426 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.712390    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"]
Jul 31 12:24:40.715860 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.715838    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:40.718776 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.718755    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 31 12:24:40.718896 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.718792    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-589zp\""
Jul 31 12:24:40.730918 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.730898    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"]
Jul 31 12:24:40.834473 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.834440    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258rd\" (UniqueName: \"kubernetes.io/projected/313c0715-e29e-4020-a8bf-f53709170b4b-kube-api-access-258rd\") pod \"kserve-controller-manager-d6f4b65b8-fb9rn\" (UID: \"313c0715-e29e-4020-a8bf-f53709170b4b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:40.834653 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.834505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/313c0715-e29e-4020-a8bf-f53709170b4b-cert\") pod \"kserve-controller-manager-d6f4b65b8-fb9rn\" (UID: \"313c0715-e29e-4020-a8bf-f53709170b4b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:40.935227 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.935196    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258rd\" (UniqueName: \"kubernetes.io/projected/313c0715-e29e-4020-a8bf-f53709170b4b-kube-api-access-258rd\") pod \"kserve-controller-manager-d6f4b65b8-fb9rn\" (UID: \"313c0715-e29e-4020-a8bf-f53709170b4b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:40.935401 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.935252    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/313c0715-e29e-4020-a8bf-f53709170b4b-cert\") pod \"kserve-controller-manager-d6f4b65b8-fb9rn\" (UID: \"313c0715-e29e-4020-a8bf-f53709170b4b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:40.935401 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:24:40.935380    2586 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 31 12:24:40.935484 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:24:40.935431    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/313c0715-e29e-4020-a8bf-f53709170b4b-cert podName:313c0715-e29e-4020-a8bf-f53709170b4b nodeName:}" failed. No retries permitted until 2026-07-31 12:24:41.435416283 +0000 UTC m=+569.422637801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/313c0715-e29e-4020-a8bf-f53709170b4b-cert") pod "kserve-controller-manager-d6f4b65b8-fb9rn" (UID: "313c0715-e29e-4020-a8bf-f53709170b4b") : secret "kserve-webhook-server-cert" not found
Jul 31 12:24:40.948335 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:40.948308    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-258rd\" (UniqueName: \"kubernetes.io/projected/313c0715-e29e-4020-a8bf-f53709170b4b-kube-api-access-258rd\") pod \"kserve-controller-manager-d6f4b65b8-fb9rn\" (UID: \"313c0715-e29e-4020-a8bf-f53709170b4b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:41.442336 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:41.442303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/313c0715-e29e-4020-a8bf-f53709170b4b-cert\") pod \"kserve-controller-manager-d6f4b65b8-fb9rn\" (UID: \"313c0715-e29e-4020-a8bf-f53709170b4b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:41.444585 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:41.444564    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/313c0715-e29e-4020-a8bf-f53709170b4b-cert\") pod \"kserve-controller-manager-d6f4b65b8-fb9rn\" (UID: \"313c0715-e29e-4020-a8bf-f53709170b4b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:41.629327 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:41.629281    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:41.750990 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:41.750961    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"]
Jul 31 12:24:41.753931 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:24:41.753906    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313c0715_e29e_4020_a8bf_f53709170b4b.slice/crio-29389177895ea3397428a91d5f641f6c34d7657d270ed8626c1abafee2bfc7ce WatchSource:0}: Error finding container 29389177895ea3397428a91d5f641f6c34d7657d270ed8626c1abafee2bfc7ce: Status 404 returned error can't find the container with id 29389177895ea3397428a91d5f641f6c34d7657d270ed8626c1abafee2bfc7ce
Jul 31 12:24:42.347433 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:42.347399    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn" event={"ID":"313c0715-e29e-4020-a8bf-f53709170b4b","Type":"ContainerStarted","Data":"29389177895ea3397428a91d5f641f6c34d7657d270ed8626c1abafee2bfc7ce"}
Jul 31 12:24:44.267623 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.267554    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"]
Jul 31 12:24:44.267967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.267730    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78" podUID="d54ec1b9-b258-468a-933f-a12feb4acd55" containerName="discovery" containerID="cri-o://cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b" gracePeriod=30
Jul 31 12:24:44.361418 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.361392    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn" event={"ID":"313c0715-e29e-4020-a8bf-f53709170b4b","Type":"ContainerStarted","Data":"28dc2a77f171caf72db86132b34adf61e74433d5dd1b4d924c06930d6c6154c5"}
Jul 31 12:24:44.361579 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.361517    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:24:44.378703 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.378656    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn" podStartSLOduration=2.196555868 podStartE2EDuration="4.378640152s" totalImagesPullingTime="2.182084284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:24:44.377220491 +0000 UTC m=+572.364442031" watchObservedRunningTime="2026-07-31 12:24:44.378640152 +0000 UTC m=+572.365861691"
Jul 31 12:24:44.506089 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.506067    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:44.573567 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.573504    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-dns-cert\") pod \"d54ec1b9-b258-468a-933f-a12feb4acd55\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") "
Jul 31 12:24:44.573567 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.573537    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-token\") pod \"d54ec1b9-b258-468a-933f-a12feb4acd55\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") "
Jul 31 12:24:44.573567 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.573565    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtmcn\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-kube-api-access-mtmcn\") pod \"d54ec1b9-b258-468a-933f-a12feb4acd55\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") "
Jul 31 12:24:44.573798 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.573588    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ec1b9-b258-468a-933f-a12feb4acd55-local-certs\") pod \"d54ec1b9-b258-468a-933f-a12feb4acd55\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") "
Jul 31 12:24:44.573798 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.573712    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-cacerts\") pod \"d54ec1b9-b258-468a-933f-a12feb4acd55\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") "
Jul 31 12:24:44.573798 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.573758    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-ca-configmap\") pod \"d54ec1b9-b258-468a-933f-a12feb4acd55\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") "
Jul 31 12:24:44.573951 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.573801    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-kubeconfig\") pod \"d54ec1b9-b258-468a-933f-a12feb4acd55\" (UID: \"d54ec1b9-b258-468a-933f-a12feb4acd55\") "
Jul 31 12:24:44.574257 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.574166    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d54ec1b9-b258-468a-933f-a12feb4acd55" (UID: "d54ec1b9-b258-468a-933f-a12feb4acd55"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:24:44.575956 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.575933    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54ec1b9-b258-468a-933f-a12feb4acd55-local-certs" (OuterVolumeSpecName: "local-certs") pod "d54ec1b9-b258-468a-933f-a12feb4acd55" (UID: "d54ec1b9-b258-468a-933f-a12feb4acd55"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:24:44.576061 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.576016    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-kube-api-access-mtmcn" (OuterVolumeSpecName: "kube-api-access-mtmcn") pod "d54ec1b9-b258-468a-933f-a12feb4acd55" (UID: "d54ec1b9-b258-468a-933f-a12feb4acd55"). InnerVolumeSpecName "kube-api-access-mtmcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:24:44.576126 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.576105    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d54ec1b9-b258-468a-933f-a12feb4acd55" (UID: "d54ec1b9-b258-468a-933f-a12feb4acd55"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:24:44.576164 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.576119    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-cacerts" (OuterVolumeSpecName: "cacerts") pod "d54ec1b9-b258-468a-933f-a12feb4acd55" (UID: "d54ec1b9-b258-468a-933f-a12feb4acd55"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:24:44.576212 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.576199    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-token" (OuterVolumeSpecName: "istio-token") pod "d54ec1b9-b258-468a-933f-a12feb4acd55" (UID: "d54ec1b9-b258-468a-933f-a12feb4acd55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:24:44.576307 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.576259    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d54ec1b9-b258-468a-933f-a12feb4acd55" (UID: "d54ec1b9-b258-468a-933f-a12feb4acd55"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:24:44.674937 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.674904    2586 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-dns-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:24:44.674937 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.674931    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:24:44.674937 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.674941    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtmcn\" (UniqueName: \"kubernetes.io/projected/d54ec1b9-b258-468a-933f-a12feb4acd55-kube-api-access-mtmcn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:24:44.675152 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.674950    2586 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ec1b9-b258-468a-933f-a12feb4acd55-local-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:24:44.675152 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.674959    2586 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-cacerts\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:24:44.675152 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.674967    2586 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-csr-ca-configmap\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:24:44.675152 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:44.674975    2586 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d54ec1b9-b258-468a-933f-a12feb4acd55-istio-kubeconfig\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:24:45.365681 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:45.365653    2586 generic.go:356] "Generic (PLEG): container finished" podID="d54ec1b9-b258-468a-933f-a12feb4acd55" containerID="cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b" exitCode=0
Jul 31 12:24:45.366068 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:45.365698    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"
Jul 31 12:24:45.366068 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:45.365743    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78" event={"ID":"d54ec1b9-b258-468a-933f-a12feb4acd55","Type":"ContainerDied","Data":"cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b"}
Jul 31 12:24:45.366068 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:45.365791    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78" event={"ID":"d54ec1b9-b258-468a-933f-a12feb4acd55","Type":"ContainerDied","Data":"f3d32d9ddc371a028742561d0113f3f5ad440b1f9f83ccb4480a3f5f87c5c172"}
Jul 31 12:24:45.366068 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:45.365817    2586 scope.go:117] "RemoveContainer" containerID="cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b"
Jul 31 12:24:45.373970 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:45.373953    2586 scope.go:117] "RemoveContainer" containerID="cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b"
Jul 31 12:24:45.374296 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:24:45.374240    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b\": container with ID starting with cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b not found: ID does not exist" containerID="cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b"
Jul 31 12:24:45.374296 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:45.374285    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b"} err="failed to get container status \"cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b\": rpc error: code = NotFound desc = could not find container \"cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b\": container with ID starting with cae70c207797b0dd14d50051eb8009ac46e3035eac68fdfc5472ae01ebbcf79b not found: ID does not exist"
Jul 31 12:24:45.389244 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:45.387650    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"]
Jul 31 12:24:45.391525 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:45.391505    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pgs78"]
Jul 31 12:24:46.484033 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:46.483996    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54ec1b9-b258-468a-933f-a12feb4acd55" path="/var/lib/kubelet/pods/d54ec1b9-b258-468a-933f-a12feb4acd55/volumes"
Jul 31 12:24:47.327368 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:47.327343    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-v6mcf"
Jul 31 12:24:47.329850 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:24:47.329829    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-9sgdg"
Jul 31 12:25:15.370524 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:15.370496    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fb9rn"
Jul 31 12:25:44.905536 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:44.905502    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-vh8kb"]
Jul 31 12:25:44.905996 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:44.905782    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54ec1b9-b258-468a-933f-a12feb4acd55" containerName="discovery"
Jul 31 12:25:44.905996 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:44.905794    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54ec1b9-b258-468a-933f-a12feb4acd55" containerName="discovery"
Jul 31 12:25:44.905996 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:44.905838    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54ec1b9-b258-468a-933f-a12feb4acd55" containerName="discovery"
Jul 31 12:25:44.908365 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:44.908349    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb"
Jul 31 12:25:44.910739 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:44.910720    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 12:25:44.910830 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:44.910722    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 12:25:44.910830 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:44.910774    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-76x4c\""
Jul 31 12:25:44.919414 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:44.919392    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-vh8kb"]
Jul 31 12:25:45.020371 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:45.020339    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qvq\" (UniqueName: \"kubernetes.io/projected/5e7c23f0-dacb-4ef0-b03b-f887654f123f-kube-api-access-k7qvq\") pod \"authorino-operator-68d88948d8-vh8kb\" (UID: \"5e7c23f0-dacb-4ef0-b03b-f887654f123f\") " pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb"
Jul 31 12:25:45.120794 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:45.120761    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qvq\" (UniqueName: \"kubernetes.io/projected/5e7c23f0-dacb-4ef0-b03b-f887654f123f-kube-api-access-k7qvq\") pod \"authorino-operator-68d88948d8-vh8kb\" (UID: \"5e7c23f0-dacb-4ef0-b03b-f887654f123f\") " pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb"
Jul 31 12:25:45.134353 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:45.134321    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qvq\" (UniqueName: \"kubernetes.io/projected/5e7c23f0-dacb-4ef0-b03b-f887654f123f-kube-api-access-k7qvq\") pod \"authorino-operator-68d88948d8-vh8kb\" (UID: \"5e7c23f0-dacb-4ef0-b03b-f887654f123f\") " pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb"
Jul 31 12:25:45.218832 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:45.218753    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb"
Jul 31 12:25:45.349876 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:45.349847    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-vh8kb"]
Jul 31 12:25:45.354185 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:25:45.354160    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7c23f0_dacb_4ef0_b03b_f887654f123f.slice/crio-04d6f469f07345cd6e1d3c0f00d9a3d4cec5f5c63ba37665ac12b2d1380daf8a WatchSource:0}: Error finding container 04d6f469f07345cd6e1d3c0f00d9a3d4cec5f5c63ba37665ac12b2d1380daf8a: Status 404 returned error can't find the container with id 04d6f469f07345cd6e1d3c0f00d9a3d4cec5f5c63ba37665ac12b2d1380daf8a
Jul 31 12:25:45.575426 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:45.575345    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb" event={"ID":"5e7c23f0-dacb-4ef0-b03b-f887654f123f","Type":"ContainerStarted","Data":"04d6f469f07345cd6e1d3c0f00d9a3d4cec5f5c63ba37665ac12b2d1380daf8a"}
Jul 31 12:25:47.330480 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:47.330445    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"]
Jul 31 12:25:47.340057 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:47.340034    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"
Jul 31 12:25:47.342712 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:47.342688    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vgddn\""
Jul 31 12:25:47.342712 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:47.342700    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 31 12:25:47.345526 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:47.345503    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"]
Jul 31 12:25:47.437095 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:47.437064    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxv2v\" (UniqueName: \"kubernetes.io/projected/28d9fb8b-f4c9-4bc7-8824-a5f26b716988-kube-api-access-lxv2v\") pod \"dns-operator-controller-manager-5985564776-xpxx2\" (UID: \"28d9fb8b-f4c9-4bc7-8824-a5f26b716988\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"
Jul 31 12:25:47.537832 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:47.537795    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxv2v\" (UniqueName: \"kubernetes.io/projected/28d9fb8b-f4c9-4bc7-8824-a5f26b716988-kube-api-access-lxv2v\") pod \"dns-operator-controller-manager-5985564776-xpxx2\" (UID: \"28d9fb8b-f4c9-4bc7-8824-a5f26b716988\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"
Jul 31 12:25:47.549526 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:47.549496    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxv2v\" (UniqueName: \"kubernetes.io/projected/28d9fb8b-f4c9-4bc7-8824-a5f26b716988-kube-api-access-lxv2v\") pod \"dns-operator-controller-manager-5985564776-xpxx2\" (UID: \"28d9fb8b-f4c9-4bc7-8824-a5f26b716988\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"
Jul 31 12:25:47.653748 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:47.653723    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"
Jul 31 12:25:48.066484 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:48.066461    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"]
Jul 31 12:25:48.069433 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:25:48.069402    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d9fb8b_f4c9_4bc7_8824_a5f26b716988.slice/crio-4d918fb7c29e5fc071aea6e18b69e11226c5a9a3b22d2152c3b2a30ae21d2686 WatchSource:0}: Error finding container 4d918fb7c29e5fc071aea6e18b69e11226c5a9a3b22d2152c3b2a30ae21d2686: Status 404 returned error can't find the container with id 4d918fb7c29e5fc071aea6e18b69e11226c5a9a3b22d2152c3b2a30ae21d2686
Jul 31 12:25:48.589430 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:48.589398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb" event={"ID":"5e7c23f0-dacb-4ef0-b03b-f887654f123f","Type":"ContainerStarted","Data":"cf9e3ff9f4d4d118835e79c2563d717a0f27933c900dafec462151ec4b118b07"}
Jul 31 12:25:48.589885 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:48.589544    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb"
Jul 31 12:25:48.590957 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:48.590903    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2" event={"ID":"28d9fb8b-f4c9-4bc7-8824-a5f26b716988","Type":"ContainerStarted","Data":"4d918fb7c29e5fc071aea6e18b69e11226c5a9a3b22d2152c3b2a30ae21d2686"}
Jul 31 12:25:48.608984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:48.608937    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb" podStartSLOduration=1.971250123 podStartE2EDuration="4.608922806s" totalImagesPullingTime="2.637672683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:25:48.608236068 +0000 UTC m=+636.595457608" watchObservedRunningTime="2026-07-31 12:25:48.608922806 +0000 UTC m=+636.596144344"
Jul 31 12:25:50.601620 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:50.601587    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2" event={"ID":"28d9fb8b-f4c9-4bc7-8824-a5f26b716988","Type":"ContainerStarted","Data":"2678146c20f48905278756354b0aab63b62258c16e11532e3fef381ea5f71b82"}
Jul 31 12:25:50.602001 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:50.601701    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"
Jul 31 12:25:50.623371 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:50.623326    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2" podStartSLOduration=1.7820720159999999 podStartE2EDuration="3.623308028s" totalImagesPullingTime="1.841236012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:25:50.621700416 +0000 UTC m=+638.608921958" watchObservedRunningTime="2026-07-31 12:25:50.623308028 +0000 UTC m=+638.610529568"
Jul 31 12:25:51.535195 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.535164    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"]
Jul 31 12:25:51.543175 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.543148    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:25:51.545740 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.545724    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-54ldl\""
Jul 31 12:25:51.553377 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.553354    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"]
Jul 31 12:25:51.673933 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.673902    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28093dd9-9e09-4507-bcb5-1ea46e773f55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" (UID: \"28093dd9-9e09-4507-bcb5-1ea46e773f55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:25:51.674279 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.673940    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn49v\" (UniqueName: \"kubernetes.io/projected/28093dd9-9e09-4507-bcb5-1ea46e773f55-kube-api-access-mn49v\") pod \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" (UID: \"28093dd9-9e09-4507-bcb5-1ea46e773f55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:25:51.774930 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.774892    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28093dd9-9e09-4507-bcb5-1ea46e773f55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" (UID: \"28093dd9-9e09-4507-bcb5-1ea46e773f55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:25:51.774930 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.774931    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn49v\" (UniqueName: \"kubernetes.io/projected/28093dd9-9e09-4507-bcb5-1ea46e773f55-kube-api-access-mn49v\") pod \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" (UID: \"28093dd9-9e09-4507-bcb5-1ea46e773f55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:25:51.775450 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.775431    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28093dd9-9e09-4507-bcb5-1ea46e773f55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" (UID: \"28093dd9-9e09-4507-bcb5-1ea46e773f55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:25:51.784479 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.784457    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn49v\" (UniqueName: \"kubernetes.io/projected/28093dd9-9e09-4507-bcb5-1ea46e773f55-kube-api-access-mn49v\") pod \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" (UID: \"28093dd9-9e09-4507-bcb5-1ea46e773f55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:25:51.853780 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.853706    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:25:51.991074 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:51.991049    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"]
Jul 31 12:25:51.993453 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:25:51.993420    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28093dd9_9e09_4507_bcb5_1ea46e773f55.slice/crio-808a1fca29a3f901e6453c5e10c2b9160a4da50e7bd28f108128d45ed92c4f35 WatchSource:0}: Error finding container 808a1fca29a3f901e6453c5e10c2b9160a4da50e7bd28f108128d45ed92c4f35: Status 404 returned error can't find the container with id 808a1fca29a3f901e6453c5e10c2b9160a4da50e7bd28f108128d45ed92c4f35
Jul 31 12:25:52.610226 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:52.610186    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh" event={"ID":"28093dd9-9e09-4507-bcb5-1ea46e773f55","Type":"ContainerStarted","Data":"808a1fca29a3f901e6453c5e10c2b9160a4da50e7bd28f108128d45ed92c4f35"}
Jul 31 12:25:53.495609 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:53.495576    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"]
Jul 31 12:25:53.498884 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:53.498862    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:25:53.504192 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:53.504168    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gzr4k\""
Jul 31 12:25:53.535924 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:53.535890    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"]
Jul 31 12:25:53.590706 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:53.590674    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxw4\" (UniqueName: \"kubernetes.io/projected/28c00b07-0f5f-4852-8da2-fe76d18377e3-kube-api-access-vhxw4\") pod \"limitador-operator-controller-manager-6c549b6d85-lgzbc\" (UID: \"28c00b07-0f5f-4852-8da2-fe76d18377e3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:25:53.691646 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:53.691612    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxw4\" (UniqueName: \"kubernetes.io/projected/28c00b07-0f5f-4852-8da2-fe76d18377e3-kube-api-access-vhxw4\") pod \"limitador-operator-controller-manager-6c549b6d85-lgzbc\" (UID: \"28c00b07-0f5f-4852-8da2-fe76d18377e3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:25:53.702177 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:53.702142    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxw4\" (UniqueName: \"kubernetes.io/projected/28c00b07-0f5f-4852-8da2-fe76d18377e3-kube-api-access-vhxw4\") pod \"limitador-operator-controller-manager-6c549b6d85-lgzbc\" (UID: \"28c00b07-0f5f-4852-8da2-fe76d18377e3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:25:53.811872 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:53.811797    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:25:53.960600 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:53.960514    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"]
Jul 31 12:25:53.963253 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:25:53.963224    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c00b07_0f5f_4852_8da2_fe76d18377e3.slice/crio-6b1ec09f7c163f54dde576dfd8655b4a11909dc4e8aceef22a5f63e008ac3158 WatchSource:0}: Error finding container 6b1ec09f7c163f54dde576dfd8655b4a11909dc4e8aceef22a5f63e008ac3158: Status 404 returned error can't find the container with id 6b1ec09f7c163f54dde576dfd8655b4a11909dc4e8aceef22a5f63e008ac3158
Jul 31 12:25:54.620132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:54.620102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc" event={"ID":"28c00b07-0f5f-4852-8da2-fe76d18377e3","Type":"ContainerStarted","Data":"6b1ec09f7c163f54dde576dfd8655b4a11909dc4e8aceef22a5f63e008ac3158"}
Jul 31 12:25:56.630438 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:56.630398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh" event={"ID":"28093dd9-9e09-4507-bcb5-1ea46e773f55","Type":"ContainerStarted","Data":"a2c6fe3665efdc6264bb9e12528b4385d88479fd0c563542a25910988abfbaaa"}
Jul 31 12:25:56.630790 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:56.630463    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:25:56.651838 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:56.651789    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh" podStartSLOduration=1.767200784 podStartE2EDuration="5.651774157s" totalImagesPullingTime="3.884573373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:25:56.649740244 +0000 UTC m=+644.636961784" watchObservedRunningTime="2026-07-31 12:25:56.651774157 +0000 UTC m=+644.638995697"
Jul 31 12:25:57.635821 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:57.635785    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc" event={"ID":"28c00b07-0f5f-4852-8da2-fe76d18377e3","Type":"ContainerStarted","Data":"35831be386510fad1b5c85930e372e8508129cdd2cd1c44611fa4e5cecd79fc9"}
Jul 31 12:25:57.636312 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:57.635974    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:25:57.653995 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:57.653935    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc" podStartSLOduration=1.955481037 podStartE2EDuration="4.653914483s" totalImagesPullingTime="2.698433446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:25:57.652180886 +0000 UTC m=+645.639402428" watchObservedRunningTime="2026-07-31 12:25:57.653914483 +0000 UTC m=+645.641136024"
Jul 31 12:25:58.575430 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.575390    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"]
Jul 31 12:25:58.580016 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.579994    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.582122 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.582101    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 31 12:25:58.582233 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.582101    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gpmcc\""
Jul 31 12:25:58.582435 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.582418    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 31 12:25:58.586820 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.586797    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"]
Jul 31 12:25:58.731948 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.731911    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/558fd764-0823-4c71-b5c1-0a756172ff45-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-q989n\" (UID: \"558fd764-0823-4c71-b5c1-0a756172ff45\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.732362 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.731972    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28rz\" (UniqueName: \"kubernetes.io/projected/558fd764-0823-4c71-b5c1-0a756172ff45-kube-api-access-q28rz\") pod \"kuadrant-console-plugin-598f86468b-q989n\" (UID: \"558fd764-0823-4c71-b5c1-0a756172ff45\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.732362 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.732037    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/558fd764-0823-4c71-b5c1-0a756172ff45-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-q989n\" (UID: \"558fd764-0823-4c71-b5c1-0a756172ff45\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.832754 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.832666    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/558fd764-0823-4c71-b5c1-0a756172ff45-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-q989n\" (UID: \"558fd764-0823-4c71-b5c1-0a756172ff45\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.832754 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.832721    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28rz\" (UniqueName: \"kubernetes.io/projected/558fd764-0823-4c71-b5c1-0a756172ff45-kube-api-access-q28rz\") pod \"kuadrant-console-plugin-598f86468b-q989n\" (UID: \"558fd764-0823-4c71-b5c1-0a756172ff45\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.832984 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.832763    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/558fd764-0823-4c71-b5c1-0a756172ff45-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-q989n\" (UID: \"558fd764-0823-4c71-b5c1-0a756172ff45\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.833476 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.833452    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/558fd764-0823-4c71-b5c1-0a756172ff45-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-q989n\" (UID: \"558fd764-0823-4c71-b5c1-0a756172ff45\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.835094 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.835071    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/558fd764-0823-4c71-b5c1-0a756172ff45-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-q989n\" (UID: \"558fd764-0823-4c71-b5c1-0a756172ff45\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.841388 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.841366    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28rz\" (UniqueName: \"kubernetes.io/projected/558fd764-0823-4c71-b5c1-0a756172ff45-kube-api-access-q28rz\") pod \"kuadrant-console-plugin-598f86468b-q989n\" (UID: \"558fd764-0823-4c71-b5c1-0a756172ff45\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:58.889994 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:58.889965    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"
Jul 31 12:25:59.021783 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:59.021761    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-q989n"]
Jul 31 12:25:59.024207 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:25:59.024180    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod558fd764_0823_4c71_b5c1_0a756172ff45.slice/crio-150f2c17ac16ecbd505c3ba0c0163e84f92fc680b1a737191ee4e4ddf92fc5df WatchSource:0}: Error finding container 150f2c17ac16ecbd505c3ba0c0163e84f92fc680b1a737191ee4e4ddf92fc5df: Status 404 returned error can't find the container with id 150f2c17ac16ecbd505c3ba0c0163e84f92fc680b1a737191ee4e4ddf92fc5df
Jul 31 12:25:59.598613 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:59.598583    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-vh8kb"
Jul 31 12:25:59.649097 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:25:59.649069    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n" event={"ID":"558fd764-0823-4c71-b5c1-0a756172ff45","Type":"ContainerStarted","Data":"150f2c17ac16ecbd505c3ba0c0163e84f92fc680b1a737191ee4e4ddf92fc5df"}
Jul 31 12:26:01.608302 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:01.608251    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-xpxx2"
Jul 31 12:26:04.681532 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:04.681490    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n" event={"ID":"558fd764-0823-4c71-b5c1-0a756172ff45","Type":"ContainerStarted","Data":"0a3905b5da88a20a2261d2bb0b6b02108c1f5973c0e93840e931f8655f5dc6e4"}
Jul 31 12:26:04.697332 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:04.697288    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-q989n" podStartSLOduration=1.686476505 podStartE2EDuration="6.697258375s" totalImagesPullingTime="5.01078187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:26:04.695912152 +0000 UTC m=+652.683133685" watchObservedRunningTime="2026-07-31 12:26:04.697258375 +0000 UTC m=+652.684479914"
Jul 31 12:26:07.638485 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:07.638458    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:26:08.645805 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:08.645779    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:26:09.404536 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.404502    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"]
Jul 31 12:26:09.404696 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.404674    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" containerName="manager" containerID="cri-o://a2c6fe3665efdc6264bb9e12528b4385d88479fd0c563542a25910988abfbaaa" gracePeriod=2
Jul 31 12:26:09.408819 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.408791    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"]
Jul 31 12:26:09.432114 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.432094    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"]
Jul 31 12:26:09.432416 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.432403    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" containerName="manager"
Jul 31 12:26:09.432416 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.432417    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" containerName="manager"
Jul 31 12:26:09.432537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.432467    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" containerName="manager"
Jul 31 12:26:09.435234 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.435218    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:09.450332 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.450312    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"]
Jul 31 12:26:09.464604 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:09.464574    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:26:09.496153 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.496129    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"]
Jul 31 12:26:09.496314 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.496295    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc" podUID="28c00b07-0f5f-4852-8da2-fe76d18377e3" containerName="manager" containerID="cri-o://35831be386510fad1b5c85930e372e8508129cdd2cd1c44611fa4e5cecd79fc9" gracePeriod=2
Jul 31 12:26:09.508195 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.508172    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"]
Jul 31 12:26:09.515513 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:09.515481    2586 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"28c00b07-0f5f-4852-8da2-fe76d18377e3\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-lgzbc\": pods \"limitador-operator-controller-manager-6c549b6d85-lgzbc\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:26:09.517480 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:09.517451    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:26:09.517760 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.517734    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"]
Jul 31 12:26:09.518043 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.518028    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c00b07-0f5f-4852-8da2-fe76d18377e3" containerName="manager"
Jul 31 12:26:09.518043 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.518043    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c00b07-0f5f-4852-8da2-fe76d18377e3" containerName="manager"
Jul 31 12:26:09.518173 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.518088    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c00b07-0f5f-4852-8da2-fe76d18377e3" containerName="manager"
Jul 31 12:26:09.520805 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.520785    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"
Jul 31 12:26:09.521107 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.521087    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbr8\" (UniqueName: \"kubernetes.io/projected/b08074c5-1602-4c9b-971b-db5f866a05b8-kube-api-access-6cbr8\") pod \"kuadrant-operator-controller-manager-58fc56757c-jjfhm\" (UID: \"b08074c5-1602-4c9b-971b-db5f866a05b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:09.521186 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.521128    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b08074c5-1602-4c9b-971b-db5f866a05b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-jjfhm\" (UID: \"b08074c5-1602-4c9b-971b-db5f866a05b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:09.522825 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:09.522800    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-lgzbc\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="28c00b07-0f5f-4852-8da2-fe76d18377e3" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:26:09.524645 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:09.524617    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:26:09.532258 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.532235    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"]
Jul 31 12:26:09.627301 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.622650    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbr8\" (UniqueName: \"kubernetes.io/projected/b08074c5-1602-4c9b-971b-db5f866a05b8-kube-api-access-6cbr8\") pod \"kuadrant-operator-controller-manager-58fc56757c-jjfhm\" (UID: \"b08074c5-1602-4c9b-971b-db5f866a05b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:09.627301 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.622755    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b08074c5-1602-4c9b-971b-db5f866a05b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-jjfhm\" (UID: \"b08074c5-1602-4c9b-971b-db5f866a05b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:09.627301 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.622879    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6h8w\" (UniqueName: \"kubernetes.io/projected/ca0e3552-5f2d-41ba-9ab9-d13684b5f047-kube-api-access-n6h8w\") pod \"limitador-operator-controller-manager-6c549b6d85-7cbm7\" (UID: \"ca0e3552-5f2d-41ba-9ab9-d13684b5f047\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"
Jul 31 12:26:09.627301 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.623120    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b08074c5-1602-4c9b-971b-db5f866a05b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-jjfhm\" (UID: \"b08074c5-1602-4c9b-971b-db5f866a05b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:09.632326 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.632299    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbr8\" (UniqueName: \"kubernetes.io/projected/b08074c5-1602-4c9b-971b-db5f866a05b8-kube-api-access-6cbr8\") pod \"kuadrant-operator-controller-manager-58fc56757c-jjfhm\" (UID: \"b08074c5-1602-4c9b-971b-db5f866a05b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:09.705891 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.705861    2586 generic.go:356] "Generic (PLEG): container finished" podID="28c00b07-0f5f-4852-8da2-fe76d18377e3" containerID="35831be386510fad1b5c85930e372e8508129cdd2cd1c44611fa4e5cecd79fc9" exitCode=0
Jul 31 12:26:09.707606 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.707582    2586 generic.go:356] "Generic (PLEG): container finished" podID="28093dd9-9e09-4507-bcb5-1ea46e773f55" containerID="a2c6fe3665efdc6264bb9e12528b4385d88479fd0c563542a25910988abfbaaa" exitCode=0
Jul 31 12:26:09.723551 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.723528    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6h8w\" (UniqueName: \"kubernetes.io/projected/ca0e3552-5f2d-41ba-9ab9-d13684b5f047-kube-api-access-n6h8w\") pod \"limitador-operator-controller-manager-6c549b6d85-7cbm7\" (UID: \"ca0e3552-5f2d-41ba-9ab9-d13684b5f047\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"
Jul 31 12:26:09.727211 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.727192    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:26:09.729513 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:09.729490    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-lgzbc\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="28c00b07-0f5f-4852-8da2-fe76d18377e3" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:26:09.729867 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.729852    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:26:09.731431 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:09.731403    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:26:09.733337 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:09.733315    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-lgzbc\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="28c00b07-0f5f-4852-8da2-fe76d18377e3" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:26:09.735517 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:09.735495    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cggjh\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:26:09.751305 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.751286    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6h8w\" (UniqueName: \"kubernetes.io/projected/ca0e3552-5f2d-41ba-9ab9-d13684b5f047-kube-api-access-n6h8w\") pod \"limitador-operator-controller-manager-6c549b6d85-7cbm7\" (UID: \"ca0e3552-5f2d-41ba-9ab9-d13684b5f047\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"
Jul 31 12:26:09.775103 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.775088    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:09.824405 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.824377    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhxw4\" (UniqueName: \"kubernetes.io/projected/28c00b07-0f5f-4852-8da2-fe76d18377e3-kube-api-access-vhxw4\") pod \"28c00b07-0f5f-4852-8da2-fe76d18377e3\" (UID: \"28c00b07-0f5f-4852-8da2-fe76d18377e3\") "
Jul 31 12:26:09.824531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.824458    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28093dd9-9e09-4507-bcb5-1ea46e773f55-extensions-socket-volume\") pod \"28093dd9-9e09-4507-bcb5-1ea46e773f55\" (UID: \"28093dd9-9e09-4507-bcb5-1ea46e773f55\") "
Jul 31 12:26:09.824531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.824493    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn49v\" (UniqueName: \"kubernetes.io/projected/28093dd9-9e09-4507-bcb5-1ea46e773f55-kube-api-access-mn49v\") pod \"28093dd9-9e09-4507-bcb5-1ea46e773f55\" (UID: \"28093dd9-9e09-4507-bcb5-1ea46e773f55\") "
Jul 31 12:26:09.825125 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.825085    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28093dd9-9e09-4507-bcb5-1ea46e773f55-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "28093dd9-9e09-4507-bcb5-1ea46e773f55" (UID: "28093dd9-9e09-4507-bcb5-1ea46e773f55"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:26:09.827127 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.827097    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c00b07-0f5f-4852-8da2-fe76d18377e3-kube-api-access-vhxw4" (OuterVolumeSpecName: "kube-api-access-vhxw4") pod "28c00b07-0f5f-4852-8da2-fe76d18377e3" (UID: "28c00b07-0f5f-4852-8da2-fe76d18377e3"). InnerVolumeSpecName "kube-api-access-vhxw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:26:09.827127 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.827117    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28093dd9-9e09-4507-bcb5-1ea46e773f55-kube-api-access-mn49v" (OuterVolumeSpecName: "kube-api-access-mn49v") pod "28093dd9-9e09-4507-bcb5-1ea46e773f55" (UID: "28093dd9-9e09-4507-bcb5-1ea46e773f55"). InnerVolumeSpecName "kube-api-access-mn49v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:26:09.876514 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.876485    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"
Jul 31 12:26:09.903581 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.903559    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"]
Jul 31 12:26:09.905333 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:26:09.905307    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08074c5_1602_4c9b_971b_db5f866a05b8.slice/crio-a075eedb714b24a99e53255e8dabc428de3b517f5021f5c44ac4d684bd1a2793 WatchSource:0}: Error finding container a075eedb714b24a99e53255e8dabc428de3b517f5021f5c44ac4d684bd1a2793: Status 404 returned error can't find the container with id a075eedb714b24a99e53255e8dabc428de3b517f5021f5c44ac4d684bd1a2793
Jul 31 12:26:09.926023 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.925997    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxw4\" (UniqueName: \"kubernetes.io/projected/28c00b07-0f5f-4852-8da2-fe76d18377e3-kube-api-access-vhxw4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:26:09.926125 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.926031    2586 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28093dd9-9e09-4507-bcb5-1ea46e773f55-extensions-socket-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:26:09.926125 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:09.926050    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn49v\" (UniqueName: \"kubernetes.io/projected/28093dd9-9e09-4507-bcb5-1ea46e773f55-kube-api-access-mn49v\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:26:10.007317 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.007295    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"]
Jul 31 12:26:10.009495 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:26:10.009469    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0e3552_5f2d_41ba_9ab9_d13684b5f047.slice/crio-69bdee3ae887892b4f458eeabb5b62e6abd487a697248b3df074862f07808396 WatchSource:0}: Error finding container 69bdee3ae887892b4f458eeabb5b62e6abd487a697248b3df074862f07808396: Status 404 returned error can't find the container with id 69bdee3ae887892b4f458eeabb5b62e6abd487a697248b3df074862f07808396
Jul 31 12:26:10.484574 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.484544    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28093dd9-9e09-4507-bcb5-1ea46e773f55" path="/var/lib/kubelet/pods/28093dd9-9e09-4507-bcb5-1ea46e773f55/volumes"
Jul 31 12:26:10.484865 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.484853    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c00b07-0f5f-4852-8da2-fe76d18377e3" path="/var/lib/kubelet/pods/28c00b07-0f5f-4852-8da2-fe76d18377e3/volumes"
Jul 31 12:26:10.713880 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.713805    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cggjh"
Jul 31 12:26:10.713880 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.713816    2586 scope.go:117] "RemoveContainer" containerID="a2c6fe3665efdc6264bb9e12528b4385d88479fd0c563542a25910988abfbaaa"
Jul 31 12:26:10.715749 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.715649    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7" event={"ID":"ca0e3552-5f2d-41ba-9ab9-d13684b5f047","Type":"ContainerStarted","Data":"e8fe15f969c90f7c614192c730b01f501e35ddf649ed03808682486d8ae30230"}
Jul 31 12:26:10.715749 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.715706    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7" event={"ID":"ca0e3552-5f2d-41ba-9ab9-d13684b5f047","Type":"ContainerStarted","Data":"69bdee3ae887892b4f458eeabb5b62e6abd487a697248b3df074862f07808396"}
Jul 31 12:26:10.715925 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.715905    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"
Jul 31 12:26:10.717793 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.717770    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm" event={"ID":"b08074c5-1602-4c9b-971b-db5f866a05b8","Type":"ContainerStarted","Data":"27cc20c01fab06db548596f8a9ad34bf127357748b0f1acfff981371929854b4"}
Jul 31 12:26:10.717897 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.717814    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm" event={"ID":"b08074c5-1602-4c9b-971b-db5f866a05b8","Type":"ContainerStarted","Data":"a075eedb714b24a99e53255e8dabc428de3b517f5021f5c44ac4d684bd1a2793"}
Jul 31 12:26:10.717897 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.717845    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:10.719172 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.719158    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-lgzbc"
Jul 31 12:26:10.722184 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.722168    2586 scope.go:117] "RemoveContainer" containerID="35831be386510fad1b5c85930e372e8508129cdd2cd1c44611fa4e5cecd79fc9"
Jul 31 12:26:10.745193 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.745151    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7" podStartSLOduration=1.7451385190000002 podStartE2EDuration="1.745138519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:26:10.743316873 +0000 UTC m=+658.730538410" watchObservedRunningTime="2026-07-31 12:26:10.745138519 +0000 UTC m=+658.732360036"
Jul 31 12:26:10.801720 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:10.801671    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm" podStartSLOduration=1.801653445 podStartE2EDuration="1.801653445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:26:10.792353116 +0000 UTC m=+658.779574658" watchObservedRunningTime="2026-07-31 12:26:10.801653445 +0000 UTC m=+658.788874962"
Jul 31 12:26:21.728230 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:21.728202    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:21.728716 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:21.728298    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7cbm7"
Jul 31 12:26:25.224965 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:25.224938    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"]
Jul 31 12:26:25.225372 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:25.225125    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm" podUID="b08074c5-1602-4c9b-971b-db5f866a05b8" containerName="manager" containerID="cri-o://27cc20c01fab06db548596f8a9ad34bf127357748b0f1acfff981371929854b4" gracePeriod=10
Jul 31 12:26:25.784408 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:25.784376    2586 generic.go:356] "Generic (PLEG): container finished" podID="b08074c5-1602-4c9b-971b-db5f866a05b8" containerID="27cc20c01fab06db548596f8a9ad34bf127357748b0f1acfff981371929854b4" exitCode=0
Jul 31 12:26:25.784583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:25.784458    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm" event={"ID":"b08074c5-1602-4c9b-971b-db5f866a05b8","Type":"ContainerDied","Data":"27cc20c01fab06db548596f8a9ad34bf127357748b0f1acfff981371929854b4"}
Jul 31 12:26:25.986403 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:25.986378    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:26.038877 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.038812    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cbr8\" (UniqueName: \"kubernetes.io/projected/b08074c5-1602-4c9b-971b-db5f866a05b8-kube-api-access-6cbr8\") pod \"b08074c5-1602-4c9b-971b-db5f866a05b8\" (UID: \"b08074c5-1602-4c9b-971b-db5f866a05b8\") "
Jul 31 12:26:26.040776 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.040752    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08074c5-1602-4c9b-971b-db5f866a05b8-kube-api-access-6cbr8" (OuterVolumeSpecName: "kube-api-access-6cbr8") pod "b08074c5-1602-4c9b-971b-db5f866a05b8" (UID: "b08074c5-1602-4c9b-971b-db5f866a05b8"). InnerVolumeSpecName "kube-api-access-6cbr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:26:26.139485 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.139456    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b08074c5-1602-4c9b-971b-db5f866a05b8-extensions-socket-volume\") pod \"b08074c5-1602-4c9b-971b-db5f866a05b8\" (UID: \"b08074c5-1602-4c9b-971b-db5f866a05b8\") "
Jul 31 12:26:26.139605 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.139576    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbr8\" (UniqueName: \"kubernetes.io/projected/b08074c5-1602-4c9b-971b-db5f866a05b8-kube-api-access-6cbr8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:26:26.139792 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.139770    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08074c5-1602-4c9b-971b-db5f866a05b8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b08074c5-1602-4c9b-971b-db5f866a05b8" (UID: "b08074c5-1602-4c9b-971b-db5f866a05b8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:26:26.239886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.239858    2586 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b08074c5-1602-4c9b-971b-db5f866a05b8-extensions-socket-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:26:26.789483 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.789402    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm" event={"ID":"b08074c5-1602-4c9b-971b-db5f866a05b8","Type":"ContainerDied","Data":"a075eedb714b24a99e53255e8dabc428de3b517f5021f5c44ac4d684bd1a2793"}
Jul 31 12:26:26.789483 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.789463    2586 scope.go:117] "RemoveContainer" containerID="27cc20c01fab06db548596f8a9ad34bf127357748b0f1acfff981371929854b4"
Jul 31 12:26:26.789670 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.789418    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"
Jul 31 12:26:26.808373 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.808350    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"]
Jul 31 12:26:26.813409 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:26.813388    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jjfhm"]
Jul 31 12:26:28.484242 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:28.484208    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08074c5-1602-4c9b-971b-db5f866a05b8" path="/var/lib/kubelet/pods/b08074c5-1602-4c9b-971b-db5f866a05b8/volumes"
Jul 31 12:26:41.609377 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.609348    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"]
Jul 31 12:26:41.609725 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.609631    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08074c5-1602-4c9b-971b-db5f866a05b8" containerName="manager"
Jul 31 12:26:41.609725 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.609641    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08074c5-1602-4c9b-971b-db5f866a05b8" containerName="manager"
Jul 31 12:26:41.609725 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.609696    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08074c5-1602-4c9b-971b-db5f866a05b8" containerName="manager"
Jul 31 12:26:41.611445 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.611430    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.615005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.614816    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-86grh\""
Jul 31 12:26:41.615005 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.614917    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 12:26:41.623396 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.623373    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"]
Jul 31 12:26:41.632485 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.632459    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.632611 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.632500    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.632611 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.632588    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.632728 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.632672    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.632786 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.632734    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwv8x\" (UniqueName: \"kubernetes.io/projected/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-kube-api-access-rwv8x\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.632786 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.632771    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.632874 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.632846    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.632913 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.632874    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.632913 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.632900    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733240 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733212    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733240 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733246    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwv8x\" (UniqueName: \"kubernetes.io/projected/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-kube-api-access-rwv8x\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733436    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733487    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733646 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733514    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733646 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733583    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733646 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733629    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733794 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733650    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733794 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733669    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733794 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733696    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733945 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733905    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.733996 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.733967    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.734177 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.734156    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.735714 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.735694    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.735931 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.735912    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.740533 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.740501    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwv8x\" (UniqueName: \"kubernetes.io/projected/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-kube-api-access-rwv8x\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.740650 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.740624    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4e5b62-ec42-4c74-a626-65a8f9f2260e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d2l22\" (UID: \"4c4e5b62-ec42-4c74-a626-65a8f9f2260e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:41.922737 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:41.922700    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:42.052576 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:42.052547    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"]
Jul 31 12:26:42.054984 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:26:42.054958    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4e5b62_ec42_4c74_a626_65a8f9f2260e.slice/crio-70dd8f5cfc91e001344a43a6a1e3a8cd71648f5bd45ec96b8a2f726c3c837c88 WatchSource:0}: Error finding container 70dd8f5cfc91e001344a43a6a1e3a8cd71648f5bd45ec96b8a2f726c3c837c88: Status 404 returned error can't find the container with id 70dd8f5cfc91e001344a43a6a1e3a8cd71648f5bd45ec96b8a2f726c3c837c88
Jul 31 12:26:42.057190 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:42.057172    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 12:26:42.855531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:42.855495    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" event={"ID":"4c4e5b62-ec42-4c74-a626-65a8f9f2260e","Type":"ContainerStarted","Data":"70dd8f5cfc91e001344a43a6a1e3a8cd71648f5bd45ec96b8a2f726c3c837c88"}
Jul 31 12:26:44.637997 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:44.637957    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:26:44.638445 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:44.638042    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:26:44.638445 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:44.638091    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:26:44.865194 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:44.865166    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" event={"ID":"4c4e5b62-ec42-4c74-a626-65a8f9f2260e","Type":"ContainerStarted","Data":"6ae7ef91dec4b727336db4474c112b032ec41bda426b5e694bfdf186f2ad7cc6"}
Jul 31 12:26:44.887606 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:44.887510    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" podStartSLOduration=1.307159918 podStartE2EDuration="3.887493341s" totalImagesPullingTime="2.580333423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:26:44.886298051 +0000 UTC m=+692.873519591" watchObservedRunningTime="2026-07-31 12:26:44.887493341 +0000 UTC m=+692.874714937"
Jul 31 12:26:44.923610 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:44.923572    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:45.928298 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:45.928256    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:46.874695 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:46.874667    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:46.875518 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:46.875503    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:26:53.151285 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.151230    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-g4phh"]
Jul 31 12:26:53.153533 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.153518    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:26:53.156656 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.156627    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 31 12:26:53.170255 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.170233    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-g4phh"]
Jul 31 12:26:53.223086 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.223057    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-g4phh"]
Jul 31 12:26:53.223818 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.223799    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdnqj\" (UniqueName: \"kubernetes.io/projected/dc71875e-b8db-4430-844d-0c567ad1b8f2-kube-api-access-qdnqj\") pod \"limitador-limitador-859c4d96d6-g4phh\" (UID: \"dc71875e-b8db-4430-844d-0c567ad1b8f2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:26:53.223867 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.223829    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc71875e-b8db-4430-844d-0c567ad1b8f2-config-file\") pod \"limitador-limitador-859c4d96d6-g4phh\" (UID: \"dc71875e-b8db-4430-844d-0c567ad1b8f2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:26:53.324473 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.324441    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc71875e-b8db-4430-844d-0c567ad1b8f2-config-file\") pod \"limitador-limitador-859c4d96d6-g4phh\" (UID: \"dc71875e-b8db-4430-844d-0c567ad1b8f2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:26:53.324671 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.324583    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdnqj\" (UniqueName: \"kubernetes.io/projected/dc71875e-b8db-4430-844d-0c567ad1b8f2-kube-api-access-qdnqj\") pod \"limitador-limitador-859c4d96d6-g4phh\" (UID: \"dc71875e-b8db-4430-844d-0c567ad1b8f2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:26:53.325189 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.325165    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc71875e-b8db-4430-844d-0c567ad1b8f2-config-file\") pod \"limitador-limitador-859c4d96d6-g4phh\" (UID: \"dc71875e-b8db-4430-844d-0c567ad1b8f2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:26:53.333241 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.333218    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdnqj\" (UniqueName: \"kubernetes.io/projected/dc71875e-b8db-4430-844d-0c567ad1b8f2-kube-api-access-qdnqj\") pod \"limitador-limitador-859c4d96d6-g4phh\" (UID: \"dc71875e-b8db-4430-844d-0c567ad1b8f2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:26:53.464121 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.464049    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:26:53.585749 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.585728    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-g4phh"]
Jul 31 12:26:53.587683 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:26:53.587658    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc71875e_b8db_4430_844d_0c567ad1b8f2.slice/crio-a537b7e04b1c40b69b8455044aeb04f574eb7249d60bcbc776e47c24c6dba0d2 WatchSource:0}: Error finding container a537b7e04b1c40b69b8455044aeb04f574eb7249d60bcbc776e47c24c6dba0d2: Status 404 returned error can't find the container with id a537b7e04b1c40b69b8455044aeb04f574eb7249d60bcbc776e47c24c6dba0d2
Jul 31 12:26:53.893217 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.893143    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-th8kx"]
Jul 31 12:26:53.895809 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.895788    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-th8kx"
Jul 31 12:26:53.898445 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.898417    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sxpdq\""
Jul 31 12:26:53.903741 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.903676    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-th8kx"]
Jul 31 12:26:53.904738 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.904686    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh" event={"ID":"dc71875e-b8db-4430-844d-0c567ad1b8f2","Type":"ContainerStarted","Data":"a537b7e04b1c40b69b8455044aeb04f574eb7249d60bcbc776e47c24c6dba0d2"}
Jul 31 12:26:53.929357 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:53.929331    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rqbq\" (UniqueName: \"kubernetes.io/projected/121e414d-7373-4119-aa4a-e1e072ffd514-kube-api-access-9rqbq\") pod \"authorino-b54df94b4-th8kx\" (UID: \"121e414d-7373-4119-aa4a-e1e072ffd514\") " pod="kuadrant-system/authorino-b54df94b4-th8kx"
Jul 31 12:26:54.029939 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.029911    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rqbq\" (UniqueName: \"kubernetes.io/projected/121e414d-7373-4119-aa4a-e1e072ffd514-kube-api-access-9rqbq\") pod \"authorino-b54df94b4-th8kx\" (UID: \"121e414d-7373-4119-aa4a-e1e072ffd514\") " pod="kuadrant-system/authorino-b54df94b4-th8kx"
Jul 31 12:26:54.039923 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.039897    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rqbq\" (UniqueName: \"kubernetes.io/projected/121e414d-7373-4119-aa4a-e1e072ffd514-kube-api-access-9rqbq\") pod \"authorino-b54df94b4-th8kx\" (UID: \"121e414d-7373-4119-aa4a-e1e072ffd514\") " pod="kuadrant-system/authorino-b54df94b4-th8kx"
Jul 31 12:26:54.085720 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.085692    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-95ksg"]
Jul 31 12:26:54.087765 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.087750    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-95ksg"
Jul 31 12:26:54.095743 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.095714    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-95ksg"]
Jul 31 12:26:54.130947 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.130925    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82dxz\" (UniqueName: \"kubernetes.io/projected/fd98220e-747c-451d-91d7-86b33b07d2f2-kube-api-access-82dxz\") pod \"authorino-c548b5d7d-95ksg\" (UID: \"fd98220e-747c-451d-91d7-86b33b07d2f2\") " pod="kuadrant-system/authorino-c548b5d7d-95ksg"
Jul 31 12:26:54.206380 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.206357    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-th8kx"
Jul 31 12:26:54.232092 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.232068    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82dxz\" (UniqueName: \"kubernetes.io/projected/fd98220e-747c-451d-91d7-86b33b07d2f2-kube-api-access-82dxz\") pod \"authorino-c548b5d7d-95ksg\" (UID: \"fd98220e-747c-451d-91d7-86b33b07d2f2\") " pod="kuadrant-system/authorino-c548b5d7d-95ksg"
Jul 31 12:26:54.240055 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.240030    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82dxz\" (UniqueName: \"kubernetes.io/projected/fd98220e-747c-451d-91d7-86b33b07d2f2-kube-api-access-82dxz\") pod \"authorino-c548b5d7d-95ksg\" (UID: \"fd98220e-747c-451d-91d7-86b33b07d2f2\") " pod="kuadrant-system/authorino-c548b5d7d-95ksg"
Jul 31 12:26:54.325105 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.325077    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-th8kx"]
Jul 31 12:26:54.327921 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:26:54.327894    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod121e414d_7373_4119_aa4a_e1e072ffd514.slice/crio-8f7fb5bfe21de39fbcec5d813ced07de6e6fddd5cd1a5ef76fd3d470e4beb44e WatchSource:0}: Error finding container 8f7fb5bfe21de39fbcec5d813ced07de6e6fddd5cd1a5ef76fd3d470e4beb44e: Status 404 returned error can't find the container with id 8f7fb5bfe21de39fbcec5d813ced07de6e6fddd5cd1a5ef76fd3d470e4beb44e
Jul 31 12:26:54.398057 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.398032    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-95ksg"
Jul 31 12:26:54.518555 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.518459    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-95ksg"]
Jul 31 12:26:54.520910 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:26:54.520886    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd98220e_747c_451d_91d7_86b33b07d2f2.slice/crio-2a6f68cabdc95372424bf4a74b42257964d05e5bdf3f6366fd5133fbfd56263b WatchSource:0}: Error finding container 2a6f68cabdc95372424bf4a74b42257964d05e5bdf3f6366fd5133fbfd56263b: Status 404 returned error can't find the container with id 2a6f68cabdc95372424bf4a74b42257964d05e5bdf3f6366fd5133fbfd56263b
Jul 31 12:26:54.911601 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.911568    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-95ksg" event={"ID":"fd98220e-747c-451d-91d7-86b33b07d2f2","Type":"ContainerStarted","Data":"2a6f68cabdc95372424bf4a74b42257964d05e5bdf3f6366fd5133fbfd56263b"}
Jul 31 12:26:54.912828 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:54.912799    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-th8kx" event={"ID":"121e414d-7373-4119-aa4a-e1e072ffd514","Type":"ContainerStarted","Data":"8f7fb5bfe21de39fbcec5d813ced07de6e6fddd5cd1a5ef76fd3d470e4beb44e"}
Jul 31 12:26:55.920688 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:55.920645    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh" event={"ID":"dc71875e-b8db-4430-844d-0c567ad1b8f2","Type":"ContainerStarted","Data":"5ccd585a6be2d71e031948a463652e70879261cfa640f85cb12b4f3956bac107"}
Jul 31 12:26:55.921154 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:55.920895    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:26:55.938768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:55.938534    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh" podStartSLOduration=1.528915484 podStartE2EDuration="2.938513521s" totalImagesPullingTime="1.409598037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:26:55.936196575 +0000 UTC m=+703.923418115" watchObservedRunningTime="2026-07-31 12:26:55.938513521 +0000 UTC m=+703.925735062"
Jul 31 12:26:56.925782 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:56.925752    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-95ksg" event={"ID":"fd98220e-747c-451d-91d7-86b33b07d2f2","Type":"ContainerStarted","Data":"738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901"}
Jul 31 12:26:56.927193 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:56.927172    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-th8kx" event={"ID":"121e414d-7373-4119-aa4a-e1e072ffd514","Type":"ContainerStarted","Data":"44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156"}
Jul 31 12:26:56.940830 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:56.940789    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-95ksg" podStartSLOduration=1.051864716 podStartE2EDuration="2.940773847s" totalImagesPullingTime="1.888909131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:26:56.939430852 +0000 UTC m=+704.926652392" watchObservedRunningTime="2026-07-31 12:26:56.940773847 +0000 UTC m=+704.927995386"
Jul 31 12:26:56.952866 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:56.952815    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-th8kx" podStartSLOduration=1.88055119 podStartE2EDuration="3.952797288s" totalImagesPullingTime="2.072246098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:26:56.952285059 +0000 UTC m=+704.939506596" watchObservedRunningTime="2026-07-31 12:26:56.952797288 +0000 UTC m=+704.940018829"
Jul 31 12:26:56.975920 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:56.975894    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-th8kx"]
Jul 31 12:26:58.935651 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:58.935611    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-th8kx" podUID="121e414d-7373-4119-aa4a-e1e072ffd514" containerName="authorino" containerID="cri-o://44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156" gracePeriod=30
Jul 31 12:26:59.172507 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.172480    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-th8kx"
Jul 31 12:26:59.277057 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.276983    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rqbq\" (UniqueName: \"kubernetes.io/projected/121e414d-7373-4119-aa4a-e1e072ffd514-kube-api-access-9rqbq\") pod \"121e414d-7373-4119-aa4a-e1e072ffd514\" (UID: \"121e414d-7373-4119-aa4a-e1e072ffd514\") "
Jul 31 12:26:59.279055 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.279030    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121e414d-7373-4119-aa4a-e1e072ffd514-kube-api-access-9rqbq" (OuterVolumeSpecName: "kube-api-access-9rqbq") pod "121e414d-7373-4119-aa4a-e1e072ffd514" (UID: "121e414d-7373-4119-aa4a-e1e072ffd514"). InnerVolumeSpecName "kube-api-access-9rqbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:26:59.377839 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.377810    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rqbq\" (UniqueName: \"kubernetes.io/projected/121e414d-7373-4119-aa4a-e1e072ffd514-kube-api-access-9rqbq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:26:59.941208 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.941175    2586 generic.go:356] "Generic (PLEG): container finished" podID="121e414d-7373-4119-aa4a-e1e072ffd514" containerID="44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156" exitCode=0
Jul 31 12:26:59.941682 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.941214    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-th8kx"
Jul 31 12:26:59.941682 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.941284    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-th8kx" event={"ID":"121e414d-7373-4119-aa4a-e1e072ffd514","Type":"ContainerDied","Data":"44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156"}
Jul 31 12:26:59.941682 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.941331    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-th8kx" event={"ID":"121e414d-7373-4119-aa4a-e1e072ffd514","Type":"ContainerDied","Data":"8f7fb5bfe21de39fbcec5d813ced07de6e6fddd5cd1a5ef76fd3d470e4beb44e"}
Jul 31 12:26:59.941682 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.941355    2586 scope.go:117] "RemoveContainer" containerID="44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156"
Jul 31 12:26:59.949682 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.949669    2586 scope.go:117] "RemoveContainer" containerID="44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156"
Jul 31 12:26:59.949941 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:26:59.949922    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156\": container with ID starting with 44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156 not found: ID does not exist" containerID="44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156"
Jul 31 12:26:59.949981 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.949950    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156"} err="failed to get container status \"44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156\": rpc error: code = NotFound desc = could not find container \"44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156\": container with ID starting with 44887aeb5cc7c3c4cfb167d6103a704c7cf4febc191735ac82a0a3cf6bd39156 not found: ID does not exist"
Jul 31 12:26:59.960593 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.960573    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-th8kx"]
Jul 31 12:26:59.968718 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:26:59.965233    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-th8kx"]
Jul 31 12:27:00.484248 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:00.484213    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="121e414d-7373-4119-aa4a-e1e072ffd514" path="/var/lib/kubelet/pods/121e414d-7373-4119-aa4a-e1e072ffd514/volumes"
Jul 31 12:27:06.928673 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:06.928586    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:27:07.463262 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:07.463225    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-g4phh"]
Jul 31 12:27:07.463456 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:07.463412    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh" podUID="dc71875e-b8db-4430-844d-0c567ad1b8f2" containerName="limitador" containerID="cri-o://5ccd585a6be2d71e031948a463652e70879261cfa640f85cb12b4f3956bac107" gracePeriod=30
Jul 31 12:27:07.977653 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:07.977617    2586 generic.go:356] "Generic (PLEG): container finished" podID="dc71875e-b8db-4430-844d-0c567ad1b8f2" containerID="5ccd585a6be2d71e031948a463652e70879261cfa640f85cb12b4f3956bac107" exitCode=0
Jul 31 12:27:07.978010 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:07.977671    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh" event={"ID":"dc71875e-b8db-4430-844d-0c567ad1b8f2","Type":"ContainerDied","Data":"5ccd585a6be2d71e031948a463652e70879261cfa640f85cb12b4f3956bac107"}
Jul 31 12:27:08.398977 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.398954    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:27:08.450818 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.450788    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdnqj\" (UniqueName: \"kubernetes.io/projected/dc71875e-b8db-4430-844d-0c567ad1b8f2-kube-api-access-qdnqj\") pod \"dc71875e-b8db-4430-844d-0c567ad1b8f2\" (UID: \"dc71875e-b8db-4430-844d-0c567ad1b8f2\") "
Jul 31 12:27:08.450949 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.450827    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc71875e-b8db-4430-844d-0c567ad1b8f2-config-file\") pod \"dc71875e-b8db-4430-844d-0c567ad1b8f2\" (UID: \"dc71875e-b8db-4430-844d-0c567ad1b8f2\") "
Jul 31 12:27:08.451223 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.451201    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc71875e-b8db-4430-844d-0c567ad1b8f2-config-file" (OuterVolumeSpecName: "config-file") pod "dc71875e-b8db-4430-844d-0c567ad1b8f2" (UID: "dc71875e-b8db-4430-844d-0c567ad1b8f2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:27:08.452769 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.452748    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc71875e-b8db-4430-844d-0c567ad1b8f2-kube-api-access-qdnqj" (OuterVolumeSpecName: "kube-api-access-qdnqj") pod "dc71875e-b8db-4430-844d-0c567ad1b8f2" (UID: "dc71875e-b8db-4430-844d-0c567ad1b8f2"). InnerVolumeSpecName "kube-api-access-qdnqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:27:08.551849 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.551780    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdnqj\" (UniqueName: \"kubernetes.io/projected/dc71875e-b8db-4430-844d-0c567ad1b8f2-kube-api-access-qdnqj\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:27:08.551849 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.551802    2586 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc71875e-b8db-4430-844d-0c567ad1b8f2-config-file\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:27:08.982741 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.982711    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh"
Jul 31 12:27:08.982741 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.982731    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-g4phh" event={"ID":"dc71875e-b8db-4430-844d-0c567ad1b8f2","Type":"ContainerDied","Data":"a537b7e04b1c40b69b8455044aeb04f574eb7249d60bcbc776e47c24c6dba0d2"}
Jul 31 12:27:08.983189 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:08.982795    2586 scope.go:117] "RemoveContainer" containerID="5ccd585a6be2d71e031948a463652e70879261cfa640f85cb12b4f3956bac107"
Jul 31 12:27:09.000464 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:09.000439    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-g4phh"]
Jul 31 12:27:09.006911 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:09.006890    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-g4phh"]
Jul 31 12:27:10.484432 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:10.484403    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc71875e-b8db-4430-844d-0c567ad1b8f2" path="/var/lib/kubelet/pods/dc71875e-b8db-4430-844d-0c567ad1b8f2/volumes"
Jul 31 12:27:12.690562 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.690490    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"]
Jul 31 12:27:12.691021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.690976    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc71875e-b8db-4430-844d-0c567ad1b8f2" containerName="limitador"
Jul 31 12:27:12.691021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.690994    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc71875e-b8db-4430-844d-0c567ad1b8f2" containerName="limitador"
Jul 31 12:27:12.691021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.691014    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121e414d-7373-4119-aa4a-e1e072ffd514" containerName="authorino"
Jul 31 12:27:12.691021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.691024    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="121e414d-7373-4119-aa4a-e1e072ffd514" containerName="authorino"
Jul 31 12:27:12.691205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.691094    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="121e414d-7373-4119-aa4a-e1e072ffd514" containerName="authorino"
Jul 31 12:27:12.691205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.691108    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc71875e-b8db-4430-844d-0c567ad1b8f2" containerName="limitador"
Jul 31 12:27:12.695436 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.695417    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:12.697687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.697670    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 31 12:27:12.697687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.697681    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 12:27:12.697859 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.697736    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 12:27:12.698663 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.698643    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-64gff\""
Jul 31 12:27:12.700706 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.700685    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"]
Jul 31 12:27:12.785011 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.784982    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhgk7\" (UniqueName: \"kubernetes.io/projected/1b436784-bb49-45ce-a75f-d33343637676-kube-api-access-nhgk7\") pod \"postgres-69fb4594cf-7rdnc\" (UID: \"1b436784-bb49-45ce-a75f-d33343637676\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:12.785124 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.785030    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b436784-bb49-45ce-a75f-d33343637676-data\") pod \"postgres-69fb4594cf-7rdnc\" (UID: \"1b436784-bb49-45ce-a75f-d33343637676\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:12.885753 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.885720    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhgk7\" (UniqueName: \"kubernetes.io/projected/1b436784-bb49-45ce-a75f-d33343637676-kube-api-access-nhgk7\") pod \"postgres-69fb4594cf-7rdnc\" (UID: \"1b436784-bb49-45ce-a75f-d33343637676\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:12.885913 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.885764    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b436784-bb49-45ce-a75f-d33343637676-data\") pod \"postgres-69fb4594cf-7rdnc\" (UID: \"1b436784-bb49-45ce-a75f-d33343637676\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:12.886091 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.886076    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b436784-bb49-45ce-a75f-d33343637676-data\") pod \"postgres-69fb4594cf-7rdnc\" (UID: \"1b436784-bb49-45ce-a75f-d33343637676\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:12.893626 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:12.893600    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhgk7\" (UniqueName: \"kubernetes.io/projected/1b436784-bb49-45ce-a75f-d33343637676-kube-api-access-nhgk7\") pod \"postgres-69fb4594cf-7rdnc\" (UID: \"1b436784-bb49-45ce-a75f-d33343637676\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:13.007511 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:13.007445    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:13.132313 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:13.132283    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"]
Jul 31 12:27:13.135007 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:27:13.134979    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b436784_bb49_45ce_a75f_d33343637676.slice/crio-53837a5af4066987702c27d9eeae3fe3e80329e0bf38feda4cfacc216c3ccf75 WatchSource:0}: Error finding container 53837a5af4066987702c27d9eeae3fe3e80329e0bf38feda4cfacc216c3ccf75: Status 404 returned error can't find the container with id 53837a5af4066987702c27d9eeae3fe3e80329e0bf38feda4cfacc216c3ccf75
Jul 31 12:27:14.008918 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:14.008870    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc" event={"ID":"1b436784-bb49-45ce-a75f-d33343637676","Type":"ContainerStarted","Data":"53837a5af4066987702c27d9eeae3fe3e80329e0bf38feda4cfacc216c3ccf75"}
Jul 31 12:27:19.032578 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:19.032542    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc" event={"ID":"1b436784-bb49-45ce-a75f-d33343637676","Type":"ContainerStarted","Data":"7b28b2611a1011974e66b36e80a3b4ab0d08f237778216799eb981dcd2a456f9"}
Jul 31 12:27:19.033034 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:19.032677    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:19.049279 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:19.049218    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc" podStartSLOduration=2.18181586 podStartE2EDuration="7.049199045s" totalImagesPullingTime="4.867383185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:27:19.046933726 +0000 UTC m=+727.034155291" watchObservedRunningTime="2026-07-31 12:27:19.049199045 +0000 UTC m=+727.036420586"
Jul 31 12:27:25.064594 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.064566    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7rdnc"
Jul 31 12:27:25.617625 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.617592    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9fbp6"]
Jul 31 12:27:25.623105 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.623089    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9fbp6"
Jul 31 12:27:25.627647 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.627596    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9fbp6"]
Jul 31 12:27:25.801963 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.801929    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8z5\" (UniqueName: \"kubernetes.io/projected/133b8dad-5779-4006-bd4c-74b6da4f9b1d-kube-api-access-qb8z5\") pod \"authorino-5dc7ccb5f-9fbp6\" (UID: \"133b8dad-5779-4006-bd4c-74b6da4f9b1d\") " pod="kuadrant-system/authorino-5dc7ccb5f-9fbp6"
Jul 31 12:27:25.876746 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.876669    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9fbp6"]
Jul 31 12:27:25.876931 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:27:25.876912    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qb8z5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-9fbp6" podUID="133b8dad-5779-4006-bd4c-74b6da4f9b1d"
Jul 31 12:27:25.903101 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.903073    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8z5\" (UniqueName: \"kubernetes.io/projected/133b8dad-5779-4006-bd4c-74b6da4f9b1d-kube-api-access-qb8z5\") pod \"authorino-5dc7ccb5f-9fbp6\" (UID: \"133b8dad-5779-4006-bd4c-74b6da4f9b1d\") " pod="kuadrant-system/authorino-5dc7ccb5f-9fbp6"
Jul 31 12:27:25.905617 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.905581    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8bdcf4486-mdwjl"]
Jul 31 12:27:25.909620 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.909591    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:27:25.911736 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.911717    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 31 12:27:25.912489 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.912464    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8z5\" (UniqueName: \"kubernetes.io/projected/133b8dad-5779-4006-bd4c-74b6da4f9b1d-kube-api-access-qb8z5\") pod \"authorino-5dc7ccb5f-9fbp6\" (UID: \"133b8dad-5779-4006-bd4c-74b6da4f9b1d\") " pod="kuadrant-system/authorino-5dc7ccb5f-9fbp6"
Jul 31 12:27:25.916892 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:25.916872    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8bdcf4486-mdwjl"]
Jul 31 12:27:26.004252 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.004222    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3792608-9414-4dd7-a04d-7952227bd887-tls-cert\") pod \"authorino-8bdcf4486-mdwjl\" (UID: \"e3792608-9414-4dd7-a04d-7952227bd887\") " pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:27:26.004431 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.004280    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9fl7\" (UniqueName: \"kubernetes.io/projected/e3792608-9414-4dd7-a04d-7952227bd887-kube-api-access-f9fl7\") pod \"authorino-8bdcf4486-mdwjl\" (UID: \"e3792608-9414-4dd7-a04d-7952227bd887\") " pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:27:26.060293 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.060252    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9fbp6"
Jul 31 12:27:26.067441 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.067417    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9fbp6"
Jul 31 12:27:26.105258 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.105223    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3792608-9414-4dd7-a04d-7952227bd887-tls-cert\") pod \"authorino-8bdcf4486-mdwjl\" (UID: \"e3792608-9414-4dd7-a04d-7952227bd887\") " pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:27:26.105475 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.105299    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9fl7\" (UniqueName: \"kubernetes.io/projected/e3792608-9414-4dd7-a04d-7952227bd887-kube-api-access-f9fl7\") pod \"authorino-8bdcf4486-mdwjl\" (UID: \"e3792608-9414-4dd7-a04d-7952227bd887\") " pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:27:26.107687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.107665    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3792608-9414-4dd7-a04d-7952227bd887-tls-cert\") pod \"authorino-8bdcf4486-mdwjl\" (UID: \"e3792608-9414-4dd7-a04d-7952227bd887\") " pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:27:26.112902 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.112870    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9fl7\" (UniqueName: \"kubernetes.io/projected/e3792608-9414-4dd7-a04d-7952227bd887-kube-api-access-f9fl7\") pod \"authorino-8bdcf4486-mdwjl\" (UID: \"e3792608-9414-4dd7-a04d-7952227bd887\") " pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:27:26.206612 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.206583    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb8z5\" (UniqueName: \"kubernetes.io/projected/133b8dad-5779-4006-bd4c-74b6da4f9b1d-kube-api-access-qb8z5\") pod \"133b8dad-5779-4006-bd4c-74b6da4f9b1d\" (UID: \"133b8dad-5779-4006-bd4c-74b6da4f9b1d\") "
Jul 31 12:27:26.208700 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.208675    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133b8dad-5779-4006-bd4c-74b6da4f9b1d-kube-api-access-qb8z5" (OuterVolumeSpecName: "kube-api-access-qb8z5") pod "133b8dad-5779-4006-bd4c-74b6da4f9b1d" (UID: "133b8dad-5779-4006-bd4c-74b6da4f9b1d"). InnerVolumeSpecName "kube-api-access-qb8z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:27:26.226522 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.226497    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:27:26.307719 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.307689    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb8z5\" (UniqueName: \"kubernetes.io/projected/133b8dad-5779-4006-bd4c-74b6da4f9b1d-kube-api-access-qb8z5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:27:26.352873 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:26.352833    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8bdcf4486-mdwjl"]
Jul 31 12:27:26.354901 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:27:26.354875    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3792608_9414_4dd7_a04d_7952227bd887.slice/crio-3df04f06ecbbde013d539e72f4f1aa387874e67cdac5de29d8bcf5844cc9c532 WatchSource:0}: Error finding container 3df04f06ecbbde013d539e72f4f1aa387874e67cdac5de29d8bcf5844cc9c532: Status 404 returned error can't find the container with id 3df04f06ecbbde013d539e72f4f1aa387874e67cdac5de29d8bcf5844cc9c532
Jul 31 12:27:27.066027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.065948    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9fbp6"
Jul 31 12:27:27.066027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.065946    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bdcf4486-mdwjl" event={"ID":"e3792608-9414-4dd7-a04d-7952227bd887","Type":"ContainerStarted","Data":"89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67"}
Jul 31 12:27:27.066027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.066001    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bdcf4486-mdwjl" event={"ID":"e3792608-9414-4dd7-a04d-7952227bd887","Type":"ContainerStarted","Data":"3df04f06ecbbde013d539e72f4f1aa387874e67cdac5de29d8bcf5844cc9c532"}
Jul 31 12:27:27.080386 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.080340    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8bdcf4486-mdwjl" podStartSLOduration=1.630950346 podStartE2EDuration="2.080319512s" totalImagesPullingTime="449.369166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:27:27.079766896 +0000 UTC m=+735.066988433" watchObservedRunningTime="2026-07-31 12:27:27.080319512 +0000 UTC m=+735.067541043"
Jul 31 12:27:27.102478 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.102447    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9fbp6"]
Jul 31 12:27:27.105757 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.105733    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-95ksg"]
Jul 31 12:27:27.105956 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.105924    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-95ksg" podUID="fd98220e-747c-451d-91d7-86b33b07d2f2" containerName="authorino" containerID="cri-o://738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901" gracePeriod=30
Jul 31 12:27:27.113797 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.112803    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9fbp6"]
Jul 31 12:27:27.354001 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.353980    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-95ksg"
Jul 31 12:27:27.521441 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.521413    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82dxz\" (UniqueName: \"kubernetes.io/projected/fd98220e-747c-451d-91d7-86b33b07d2f2-kube-api-access-82dxz\") pod \"fd98220e-747c-451d-91d7-86b33b07d2f2\" (UID: \"fd98220e-747c-451d-91d7-86b33b07d2f2\") "
Jul 31 12:27:27.523512 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.523487    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd98220e-747c-451d-91d7-86b33b07d2f2-kube-api-access-82dxz" (OuterVolumeSpecName: "kube-api-access-82dxz") pod "fd98220e-747c-451d-91d7-86b33b07d2f2" (UID: "fd98220e-747c-451d-91d7-86b33b07d2f2"). InnerVolumeSpecName "kube-api-access-82dxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:27:27.622191 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:27.622085    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82dxz\" (UniqueName: \"kubernetes.io/projected/fd98220e-747c-451d-91d7-86b33b07d2f2-kube-api-access-82dxz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:27:28.070825 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.070794    2586 generic.go:356] "Generic (PLEG): container finished" podID="fd98220e-747c-451d-91d7-86b33b07d2f2" containerID="738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901" exitCode=0
Jul 31 12:27:28.071000 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.070838    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-95ksg"
Jul 31 12:27:28.071000 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.070889    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-95ksg" event={"ID":"fd98220e-747c-451d-91d7-86b33b07d2f2","Type":"ContainerDied","Data":"738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901"}
Jul 31 12:27:28.071000 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.070937    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-95ksg" event={"ID":"fd98220e-747c-451d-91d7-86b33b07d2f2","Type":"ContainerDied","Data":"2a6f68cabdc95372424bf4a74b42257964d05e5bdf3f6366fd5133fbfd56263b"}
Jul 31 12:27:28.071000 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.070959    2586 scope.go:117] "RemoveContainer" containerID="738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901"
Jul 31 12:27:28.079629 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.079613    2586 scope.go:117] "RemoveContainer" containerID="738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901"
Jul 31 12:27:28.079884 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:27:28.079867    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901\": container with ID starting with 738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901 not found: ID does not exist" containerID="738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901"
Jul 31 12:27:28.079936 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.079893    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901"} err="failed to get container status \"738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901\": rpc error: code = NotFound desc = could not find container \"738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901\": container with ID starting with 738f25c33adc24fb0a623101e0c1a2a0a261096e95ed6356cc27c1ce04441901 not found: ID does not exist"
Jul 31 12:27:28.096300 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.096261    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-95ksg"]
Jul 31 12:27:28.098299 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.098279    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-95ksg"]
Jul 31 12:27:28.483989 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.483958    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="133b8dad-5779-4006-bd4c-74b6da4f9b1d" path="/var/lib/kubelet/pods/133b8dad-5779-4006-bd4c-74b6da4f9b1d/volumes"
Jul 31 12:27:28.484179 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:28.484168    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd98220e-747c-451d-91d7-86b33b07d2f2" path="/var/lib/kubelet/pods/fd98220e-747c-451d-91d7-86b33b07d2f2/volumes"
Jul 31 12:27:31.788863 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.788830    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b9685c684-tp2jn"]
Jul 31 12:27:31.789308 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.789284    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd98220e-747c-451d-91d7-86b33b07d2f2" containerName="authorino"
Jul 31 12:27:31.789308 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.789300    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd98220e-747c-451d-91d7-86b33b07d2f2" containerName="authorino"
Jul 31 12:27:31.789413 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.789383    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd98220e-747c-451d-91d7-86b33b07d2f2" containerName="authorino"
Jul 31 12:27:31.794788 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.794771    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:31.799374 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.799355    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 31 12:27:31.799489 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.799412    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 31 12:27:31.799489 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.799421    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-x7jpf\""
Jul 31 12:27:31.807978 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.807957    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9685c684-tp2jn"]
Jul 31 12:27:31.856068 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.856041    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbfsz\" (UniqueName: \"kubernetes.io/projected/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-kube-api-access-dbfsz\") pod \"maas-controller-7b9685c684-tp2jn\" (UID: \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\") " pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:31.856222 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.856109    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-webhook-cert\") pod \"maas-controller-7b9685c684-tp2jn\" (UID: \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\") " pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:31.957364 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.957336    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-webhook-cert\") pod \"maas-controller-7b9685c684-tp2jn\" (UID: \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\") " pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:31.957538 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.957392    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfsz\" (UniqueName: \"kubernetes.io/projected/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-kube-api-access-dbfsz\") pod \"maas-controller-7b9685c684-tp2jn\" (UID: \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\") " pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:31.957538 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:27:31.957504    2586 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 31 12:27:31.957645 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:27:31.957573    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-webhook-cert podName:fa34468c-d8ba-4d84-8f0c-0eb6908e6102 nodeName:}" failed. No retries permitted until 2026-07-31 12:27:32.457553328 +0000 UTC m=+740.444774850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-webhook-cert") pod "maas-controller-7b9685c684-tp2jn" (UID: "fa34468c-d8ba-4d84-8f0c-0eb6908e6102") : secret "maas-controller-webhook-cert" not found
Jul 31 12:27:31.966482 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.966457    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c67bf5868-jz4m8"]
Jul 31 12:27:31.970183 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.970159    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:31.972180 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.972160    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfsz\" (UniqueName: \"kubernetes.io/projected/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-kube-api-access-dbfsz\") pod \"maas-controller-7b9685c684-tp2jn\" (UID: \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\") " pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:31.981389 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:31.981366    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-jz4m8"]
Jul 31 12:27:32.058536 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.058448    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svbbt\" (UniqueName: \"kubernetes.io/projected/38c89a0f-fe55-4960-bd37-378ad36e0802-kube-api-access-svbbt\") pod \"maas-controller-7c67bf5868-jz4m8\" (UID: \"38c89a0f-fe55-4960-bd37-378ad36e0802\") " pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:32.058676 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.058545    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38c89a0f-fe55-4960-bd37-378ad36e0802-webhook-cert\") pod \"maas-controller-7c67bf5868-jz4m8\" (UID: \"38c89a0f-fe55-4960-bd37-378ad36e0802\") " pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:32.159206 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.159177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svbbt\" (UniqueName: \"kubernetes.io/projected/38c89a0f-fe55-4960-bd37-378ad36e0802-kube-api-access-svbbt\") pod \"maas-controller-7c67bf5868-jz4m8\" (UID: \"38c89a0f-fe55-4960-bd37-378ad36e0802\") " pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:32.159382 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.159228    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38c89a0f-fe55-4960-bd37-378ad36e0802-webhook-cert\") pod \"maas-controller-7c67bf5868-jz4m8\" (UID: \"38c89a0f-fe55-4960-bd37-378ad36e0802\") " pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:32.161492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.161475    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38c89a0f-fe55-4960-bd37-378ad36e0802-webhook-cert\") pod \"maas-controller-7c67bf5868-jz4m8\" (UID: \"38c89a0f-fe55-4960-bd37-378ad36e0802\") " pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:32.167882 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.167861    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svbbt\" (UniqueName: \"kubernetes.io/projected/38c89a0f-fe55-4960-bd37-378ad36e0802-kube-api-access-svbbt\") pod \"maas-controller-7c67bf5868-jz4m8\" (UID: \"38c89a0f-fe55-4960-bd37-378ad36e0802\") " pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:32.288697 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.288661    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:32.414435 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.414363    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-jz4m8"]
Jul 31 12:27:32.416684 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:27:32.416657    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c89a0f_fe55_4960_bd37_378ad36e0802.slice/crio-6749630376b97d2ada8b5b03535d0469c6eb5e2961d7ac8f426d7f5fcfc35273 WatchSource:0}: Error finding container 6749630376b97d2ada8b5b03535d0469c6eb5e2961d7ac8f426d7f5fcfc35273: Status 404 returned error can't find the container with id 6749630376b97d2ada8b5b03535d0469c6eb5e2961d7ac8f426d7f5fcfc35273
Jul 31 12:27:32.462503 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.462471    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-webhook-cert\") pod \"maas-controller-7b9685c684-tp2jn\" (UID: \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\") " pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:32.464836 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.464811    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-webhook-cert\") pod \"maas-controller-7b9685c684-tp2jn\" (UID: \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\") " pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:32.704609 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.704581    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:32.835817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:32.835737    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9685c684-tp2jn"]
Jul 31 12:27:32.838343 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:27:32.838313    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa34468c_d8ba_4d84_8f0c_0eb6908e6102.slice/crio-b0011bccc412997a35fcc2e16f3493ab0899194795bd9457d3ce8b9f571dd5b8 WatchSource:0}: Error finding container b0011bccc412997a35fcc2e16f3493ab0899194795bd9457d3ce8b9f571dd5b8: Status 404 returned error can't find the container with id b0011bccc412997a35fcc2e16f3493ab0899194795bd9457d3ce8b9f571dd5b8
Jul 31 12:27:33.097257 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:33.097164    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-jz4m8" event={"ID":"38c89a0f-fe55-4960-bd37-378ad36e0802","Type":"ContainerStarted","Data":"6749630376b97d2ada8b5b03535d0469c6eb5e2961d7ac8f426d7f5fcfc35273"}
Jul 31 12:27:33.098995 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:33.098960    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9685c684-tp2jn" event={"ID":"fa34468c-d8ba-4d84-8f0c-0eb6908e6102","Type":"ContainerStarted","Data":"b0011bccc412997a35fcc2e16f3493ab0899194795bd9457d3ce8b9f571dd5b8"}
Jul 31 12:27:35.109092 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:35.109010    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9685c684-tp2jn" event={"ID":"fa34468c-d8ba-4d84-8f0c-0eb6908e6102","Type":"ContainerStarted","Data":"3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3"}
Jul 31 12:27:35.109498 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:35.109101    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:35.110575 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:35.110547    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-jz4m8" event={"ID":"38c89a0f-fe55-4960-bd37-378ad36e0802","Type":"ContainerStarted","Data":"a5d3dc3dc0d7e32a4a24367807b46b65a7636ca42d6967c456e246293e7767c6"}
Jul 31 12:27:35.110684 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:35.110604    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:35.129453 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:35.129411    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b9685c684-tp2jn" podStartSLOduration=2.124972099 podStartE2EDuration="4.129397668s" totalImagesPullingTime="2.004425569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:27:35.128481849 +0000 UTC m=+743.115703390" watchObservedRunningTime="2026-07-31 12:27:35.129397668 +0000 UTC m=+743.116619207"
Jul 31 12:27:35.152045 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:35.151995    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c67bf5868-jz4m8" podStartSLOduration=1.73236415 podStartE2EDuration="4.151975849s" totalImagesPullingTime="2.419611699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:27:35.151200982 +0000 UTC m=+743.138422525" watchObservedRunningTime="2026-07-31 12:27:35.151975849 +0000 UTC m=+743.139197389"
Jul 31 12:27:46.117337 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.117307    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:46.117742 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.117375    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:27:46.176227 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.176194    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9685c684-tp2jn"]
Jul 31 12:27:46.176451 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.176426    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b9685c684-tp2jn" podUID="fa34468c-d8ba-4d84-8f0c-0eb6908e6102" containerName="manager" containerID="cri-o://3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3" gracePeriod=30
Jul 31 12:27:46.420940 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.420919    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:46.476375 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.476346    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-webhook-cert\") pod \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\" (UID: \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\") "
Jul 31 12:27:46.476588 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.476382    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbfsz\" (UniqueName: \"kubernetes.io/projected/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-kube-api-access-dbfsz\") pod \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\" (UID: \"fa34468c-d8ba-4d84-8f0c-0eb6908e6102\") "
Jul 31 12:27:46.478559 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.478527    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fa34468c-d8ba-4d84-8f0c-0eb6908e6102" (UID: "fa34468c-d8ba-4d84-8f0c-0eb6908e6102"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:27:46.478714 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.478688    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-kube-api-access-dbfsz" (OuterVolumeSpecName: "kube-api-access-dbfsz") pod "fa34468c-d8ba-4d84-8f0c-0eb6908e6102" (UID: "fa34468c-d8ba-4d84-8f0c-0eb6908e6102"). InnerVolumeSpecName "kube-api-access-dbfsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:27:46.577244 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.577212    2586 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-webhook-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:27:46.577502 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:46.577483    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbfsz\" (UniqueName: \"kubernetes.io/projected/fa34468c-d8ba-4d84-8f0c-0eb6908e6102-kube-api-access-dbfsz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:27:47.161236 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:47.161204    2586 generic.go:356] "Generic (PLEG): container finished" podID="fa34468c-d8ba-4d84-8f0c-0eb6908e6102" containerID="3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3" exitCode=0
Jul 31 12:27:47.161690 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:47.161249    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9685c684-tp2jn"
Jul 31 12:27:47.161690 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:47.161287    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9685c684-tp2jn" event={"ID":"fa34468c-d8ba-4d84-8f0c-0eb6908e6102","Type":"ContainerDied","Data":"3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3"}
Jul 31 12:27:47.161690 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:47.161329    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9685c684-tp2jn" event={"ID":"fa34468c-d8ba-4d84-8f0c-0eb6908e6102","Type":"ContainerDied","Data":"b0011bccc412997a35fcc2e16f3493ab0899194795bd9457d3ce8b9f571dd5b8"}
Jul 31 12:27:47.161690 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:47.161350    2586 scope.go:117] "RemoveContainer" containerID="3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3"
Jul 31 12:27:47.169356 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:47.169341    2586 scope.go:117] "RemoveContainer" containerID="3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3"
Jul 31 12:27:47.169584 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:27:47.169565    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3\": container with ID starting with 3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3 not found: ID does not exist" containerID="3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3"
Jul 31 12:27:47.169628 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:47.169592    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3"} err="failed to get container status \"3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3\": rpc error: code = NotFound desc = could not find container \"3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3\": container with ID starting with 3a86cb49777fd860b50b377101095278c91ccc87f8e62a66b065c39db81778c3 not found: ID does not exist"
Jul 31 12:27:47.177127 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:47.177103    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9685c684-tp2jn"]
Jul 31 12:27:47.181189 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:47.181171    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b9685c684-tp2jn"]
Jul 31 12:27:48.484608 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:27:48.484575    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa34468c-d8ba-4d84-8f0c-0eb6908e6102" path="/var/lib/kubelet/pods/fa34468c-d8ba-4d84-8f0c-0eb6908e6102/volumes"
Jul 31 12:28:07.690457 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.690426    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"]
Jul 31 12:28:07.690905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.690728    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa34468c-d8ba-4d84-8f0c-0eb6908e6102" containerName="manager"
Jul 31 12:28:07.690905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.690753    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa34468c-d8ba-4d84-8f0c-0eb6908e6102" containerName="manager"
Jul 31 12:28:07.690905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.690815    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa34468c-d8ba-4d84-8f0c-0eb6908e6102" containerName="manager"
Jul 31 12:28:07.692652 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.692635    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:07.696609 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.696590    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5fx5\""
Jul 31 12:28:07.696765 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.696750    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 31 12:28:07.707217 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.707198    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"]
Jul 31 12:28:07.751103 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.751075    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e162864-1ba6-4afe-97ee-b6ec98606d1c-openshift-service-ca\") pod \"maas-api-b9d74c676-4zjcp\" (UID: \"3e162864-1ba6-4afe-97ee-b6ec98606d1c\") " pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:07.751251 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.751115    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e162864-1ba6-4afe-97ee-b6ec98606d1c-maas-api-tls\") pod \"maas-api-b9d74c676-4zjcp\" (UID: \"3e162864-1ba6-4afe-97ee-b6ec98606d1c\") " pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:07.751251 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.751185    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkf9q\" (UniqueName: \"kubernetes.io/projected/3e162864-1ba6-4afe-97ee-b6ec98606d1c-kube-api-access-qkf9q\") pod \"maas-api-b9d74c676-4zjcp\" (UID: \"3e162864-1ba6-4afe-97ee-b6ec98606d1c\") " pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:07.852369 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.852336    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkf9q\" (UniqueName: \"kubernetes.io/projected/3e162864-1ba6-4afe-97ee-b6ec98606d1c-kube-api-access-qkf9q\") pod \"maas-api-b9d74c676-4zjcp\" (UID: \"3e162864-1ba6-4afe-97ee-b6ec98606d1c\") " pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:07.852536 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.852413    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e162864-1ba6-4afe-97ee-b6ec98606d1c-openshift-service-ca\") pod \"maas-api-b9d74c676-4zjcp\" (UID: \"3e162864-1ba6-4afe-97ee-b6ec98606d1c\") " pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:07.852536 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.852453    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e162864-1ba6-4afe-97ee-b6ec98606d1c-maas-api-tls\") pod \"maas-api-b9d74c676-4zjcp\" (UID: \"3e162864-1ba6-4afe-97ee-b6ec98606d1c\") " pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:07.853204 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.853181    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e162864-1ba6-4afe-97ee-b6ec98606d1c-openshift-service-ca\") pod \"maas-api-b9d74c676-4zjcp\" (UID: \"3e162864-1ba6-4afe-97ee-b6ec98606d1c\") " pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:07.855088 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.855067    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e162864-1ba6-4afe-97ee-b6ec98606d1c-maas-api-tls\") pod \"maas-api-b9d74c676-4zjcp\" (UID: \"3e162864-1ba6-4afe-97ee-b6ec98606d1c\") " pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:07.860844 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:07.860822    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkf9q\" (UniqueName: \"kubernetes.io/projected/3e162864-1ba6-4afe-97ee-b6ec98606d1c-kube-api-access-qkf9q\") pod \"maas-api-b9d74c676-4zjcp\" (UID: \"3e162864-1ba6-4afe-97ee-b6ec98606d1c\") " pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:08.002807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.002729    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:08.141149 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.141113    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"]
Jul 31 12:28:08.144169 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:28:08.144138    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e162864_1ba6_4afe_97ee_b6ec98606d1c.slice/crio-e006c29fd4949d444bc6e4f27812a9fb7aeb58494716c58b51e85dbab5d705bc WatchSource:0}: Error finding container e006c29fd4949d444bc6e4f27812a9fb7aeb58494716c58b51e85dbab5d705bc: Status 404 returned error can't find the container with id e006c29fd4949d444bc6e4f27812a9fb7aeb58494716c58b51e85dbab5d705bc
Jul 31 12:28:08.209728 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.209697    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-pdrrk"]
Jul 31 12:28:08.212435 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.212419    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:08.214757 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.214738    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 31 12:28:08.214883 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.214870    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4749k\""
Jul 31 12:28:08.224035 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.224015    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-pdrrk"]
Jul 31 12:28:08.234472 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.234449    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"]
Jul 31 12:28:08.236822 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.236799    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:08.253391 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.253331    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"]
Jul 31 12:28:08.261155 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.261128    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp" event={"ID":"3e162864-1ba6-4afe-97ee-b6ec98606d1c","Type":"ContainerStarted","Data":"e006c29fd4949d444bc6e4f27812a9fb7aeb58494716c58b51e85dbab5d705bc"}
Jul 31 12:28:08.357573 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.357536    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgk69\" (UniqueName: \"kubernetes.io/projected/5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc-kube-api-access-fgk69\") pod \"payload-pre-processing-687dbb6cf6-2rx7z\" (UID: \"5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:08.357713 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.357597    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-pdrrk\" (UID: \"d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:08.357713 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.357621    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbmp\" (UniqueName: \"kubernetes.io/projected/d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2-kube-api-access-vlbmp\") pod \"payload-processing-5cc6cd444b-pdrrk\" (UID: \"d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:08.357713 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.357638    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2rx7z\" (UID: \"5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:08.458843 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.458805    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgk69\" (UniqueName: \"kubernetes.io/projected/5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc-kube-api-access-fgk69\") pod \"payload-pre-processing-687dbb6cf6-2rx7z\" (UID: \"5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:08.459012 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.458874    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-pdrrk\" (UID: \"d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:08.459012 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.458909    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbmp\" (UniqueName: \"kubernetes.io/projected/d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2-kube-api-access-vlbmp\") pod \"payload-processing-5cc6cd444b-pdrrk\" (UID: \"d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:08.459012 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.458936    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2rx7z\" (UID: \"5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:08.459651 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.459630    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-pdrrk\" (UID: \"d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:08.459730 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.459631    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2rx7z\" (UID: \"5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:08.473923 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.473895    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbmp\" (UniqueName: \"kubernetes.io/projected/d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2-kube-api-access-vlbmp\") pod \"payload-processing-5cc6cd444b-pdrrk\" (UID: \"d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:08.474034 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.473952    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgk69\" (UniqueName: \"kubernetes.io/projected/5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc-kube-api-access-fgk69\") pod \"payload-pre-processing-687dbb6cf6-2rx7z\" (UID: \"5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:08.523716 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.523659    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:08.560076 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.560035    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:08.657886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.657757    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-pdrrk"]
Jul 31 12:28:08.660417 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:28:08.660390    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a4ef6f_d103_4ce3_b14c_a5020a0c2ae2.slice/crio-c80334a808447d3c7046defb7587e336e9a5cda305ae9572b1efccba67c78e16 WatchSource:0}: Error finding container c80334a808447d3c7046defb7587e336e9a5cda305ae9572b1efccba67c78e16: Status 404 returned error can't find the container with id c80334a808447d3c7046defb7587e336e9a5cda305ae9572b1efccba67c78e16
Jul 31 12:28:08.704866 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:08.704833    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"]
Jul 31 12:28:08.708475 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:28:08.708451    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff2dd07_a0e6_4073_91d9_12e7e2bf72cc.slice/crio-26a697ea932c7596771f94c573da5f86b68c4ca8e9ca724cf80e07725f5c8e20 WatchSource:0}: Error finding container 26a697ea932c7596771f94c573da5f86b68c4ca8e9ca724cf80e07725f5c8e20: Status 404 returned error can't find the container with id 26a697ea932c7596771f94c573da5f86b68c4ca8e9ca724cf80e07725f5c8e20
Jul 31 12:28:09.270491 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:09.270391    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk" event={"ID":"d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2","Type":"ContainerStarted","Data":"c80334a808447d3c7046defb7587e336e9a5cda305ae9572b1efccba67c78e16"}
Jul 31 12:28:09.273015 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:09.272939    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z" event={"ID":"5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc","Type":"ContainerStarted","Data":"26a697ea932c7596771f94c573da5f86b68c4ca8e9ca724cf80e07725f5c8e20"}
Jul 31 12:28:11.287826 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:11.287787    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp" event={"ID":"3e162864-1ba6-4afe-97ee-b6ec98606d1c","Type":"ContainerStarted","Data":"b8742779d8a89d39204acb995bc872174cc034d11a4c1c4804a5ee1b3900b2ac"}
Jul 31 12:28:11.288228 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:11.288064    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:11.304884 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:11.304831    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp" podStartSLOduration=1.981439741 podStartE2EDuration="4.304811278s" totalImagesPullingTime="2.323371537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:28:11.302248322 +0000 UTC m=+779.289469862" watchObservedRunningTime="2026-07-31 12:28:11.304811278 +0000 UTC m=+779.292032819"
Jul 31 12:28:12.294830 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:12.294739    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk" event={"ID":"d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2","Type":"ContainerStarted","Data":"b976f972dcbdc5c5f1b920fb2f5848624b59e6f394f7dd119f090ac70acae77f"}
Jul 31 12:28:12.294830 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:12.294810    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:12.296406 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:12.296382    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z" event={"ID":"5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc","Type":"ContainerStarted","Data":"25d924c417c5b9697bca68c08f8e116baeccf8800bc84b4db772cfb334cf53e4"}
Jul 31 12:28:12.296543 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:12.296516    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:12.310462 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:12.310422    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk" podStartSLOduration=0.973316978 podStartE2EDuration="4.310408913s" totalImagesPullingTime="3.337091935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:28:12.308861425 +0000 UTC m=+780.296082964" watchObservedRunningTime="2026-07-31 12:28:12.310408913 +0000 UTC m=+780.297630453"
Jul 31 12:28:12.323952 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:12.323916    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z" podStartSLOduration=1.030189917 podStartE2EDuration="4.323903749s" totalImagesPullingTime="3.293713832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:28:12.323128426 +0000 UTC m=+780.310349965" watchObservedRunningTime="2026-07-31 12:28:12.323903749 +0000 UTC m=+780.311125289"
Jul 31 12:28:16.994479 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:16.994441    2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d2l22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": read tcp 10.132.0.2:47534->10.132.0.26:15021: read: connection reset by peer" start-of-body=
Jul 31 12:28:16.994869 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:16.994532    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" podUID="4c4e5b62-ec42-4c74-a626-65a8f9f2260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": read tcp 10.132.0.2:47534->10.132.0.26:15021: read: connection reset by peer"
Jul 31 12:28:17.301649 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:17.301570    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b9d74c676-4zjcp"
Jul 31 12:28:17.322314 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:17.322255    2586 generic.go:356] "Generic (PLEG): container finished" podID="4c4e5b62-ec42-4c74-a626-65a8f9f2260e" containerID="6ae7ef91dec4b727336db4474c112b032ec41bda426b5e694bfdf186f2ad7cc6" exitCode=0
Jul 31 12:28:17.322465 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:17.322309    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" event={"ID":"4c4e5b62-ec42-4c74-a626-65a8f9f2260e","Type":"ContainerDied","Data":"6ae7ef91dec4b727336db4474c112b032ec41bda426b5e694bfdf186f2ad7cc6"}
Jul 31 12:28:17.322703 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:17.322688    2586 scope.go:117] "RemoveContainer" containerID="6ae7ef91dec4b727336db4474c112b032ec41bda426b5e694bfdf186f2ad7cc6"
Jul 31 12:28:17.323852 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:17.323826    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:28:17.323952 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:17.323892    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:28:17.323952 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:17.323926    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:28:18.329730 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:18.329697    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" event={"ID":"4c4e5b62-ec42-4c74-a626-65a8f9f2260e","Type":"ContainerStarted","Data":"fdbd48d9567d8f28a819103e01cfef3359653cd49fb464f087ddd5db6b1d36a3"}
Jul 31 12:28:18.923526 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:18.923497    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:28:18.923729 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:18.923703    2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d2l22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Jul 31 12:28:18.923775 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:18.923759    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" podUID="4c4e5b62-ec42-4c74-a626-65a8f9f2260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 31 12:28:19.923659 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:19.923620    2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d2l22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Jul 31 12:28:19.924047 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:19.923674    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" podUID="4c4e5b62-ec42-4c74-a626-65a8f9f2260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 31 12:28:20.923949 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:20.923914    2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d2l22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Jul 31 12:28:20.924350 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:20.923982    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" podUID="4c4e5b62-ec42-4c74-a626-65a8f9f2260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 31 12:28:21.923227 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:21.923187    2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d2l22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Jul 31 12:28:21.923427 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:21.923260    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" podUID="4c4e5b62-ec42-4c74-a626-65a8f9f2260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 31 12:28:22.923178 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:22.923145    2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d2l22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Jul 31 12:28:22.923596 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:22.923207    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22" podUID="4c4e5b62-ec42-4c74-a626-65a8f9f2260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 31 12:28:23.303370 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:23.303290    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-pdrrk"
Jul 31 12:28:23.303508 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:23.303390    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2rx7z"
Jul 31 12:28:23.926381 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:23.926354    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:28:23.926735 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:23.926718    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:28:23.927437 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:23.927421    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d2l22"
Jul 31 12:28:34.340707 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.340634    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"]
Jul 31 12:28:34.344102 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.344078    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.347497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.347436    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 31 12:28:34.347497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.347446    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 31 12:28:34.347497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.347436    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vmtl2\""
Jul 31 12:28:34.347497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.347475    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 31 12:28:34.355862 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.355841    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"]
Jul 31 12:28:34.387714 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.387688    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.387827 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.387716    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c1ecf6-e394-4a9f-a88e-0b0487221521-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.387827 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.387769    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.387908 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.387821    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fp45\" (UniqueName: \"kubernetes.io/projected/a2c1ecf6-e394-4a9f-a88e-0b0487221521-kube-api-access-6fp45\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.387908 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.387850    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.387908 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.387899    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.488318 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.488287    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.488318 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.488321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c1ecf6-e394-4a9f-a88e-0b0487221521-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.488531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.488372    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.488531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.488398    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fp45\" (UniqueName: \"kubernetes.io/projected/a2c1ecf6-e394-4a9f-a88e-0b0487221521-kube-api-access-6fp45\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.488531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.488424    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.488531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.488463    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.488738 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.488621    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.488738 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.488718    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.488819 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.488785    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.490550 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.490533    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c1ecf6-e394-4a9f-a88e-0b0487221521-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.490796 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.490780    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c1ecf6-e394-4a9f-a88e-0b0487221521-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.495798 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.495774    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fp45\" (UniqueName: \"kubernetes.io/projected/a2c1ecf6-e394-4a9f-a88e-0b0487221521-kube-api-access-6fp45\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg\" (UID: \"a2c1ecf6-e394-4a9f-a88e-0b0487221521\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.657325 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.657298    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:34.803203 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:34.803168    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"]
Jul 31 12:28:34.807043 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:28:34.807012    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c1ecf6_e394_4a9f_a88e_0b0487221521.slice/crio-63cd7a584471ad708887c4284073e895e741ddaf3ce60a8c76ca61e7b0ecb0c3 WatchSource:0}: Error finding container 63cd7a584471ad708887c4284073e895e741ddaf3ce60a8c76ca61e7b0ecb0c3: Status 404 returned error can't find the container with id 63cd7a584471ad708887c4284073e895e741ddaf3ce60a8c76ca61e7b0ecb0c3
Jul 31 12:28:35.411392 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:35.411348    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg" event={"ID":"a2c1ecf6-e394-4a9f-a88e-0b0487221521","Type":"ContainerStarted","Data":"63cd7a584471ad708887c4284073e895e741ddaf3ce60a8c76ca61e7b0ecb0c3"}
Jul 31 12:28:36.418120 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:36.418086    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg" event={"ID":"a2c1ecf6-e394-4a9f-a88e-0b0487221521","Type":"ContainerStarted","Data":"9d74615de1fc97a9f8720cb94685a964eee4a06775ef231c9e71d3852aba06d6"}
Jul 31 12:28:36.434841 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:36.434774    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg" podStartSLOduration=1.328250057 podStartE2EDuration="2.434759635s" totalImagesPullingTime="1.106509578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:28:36.433427919 +0000 UTC m=+804.420649462" watchObservedRunningTime="2026-07-31 12:28:36.434759635 +0000 UTC m=+804.421981176"
Jul 31 12:28:44.657563 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:44.657520    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:44.657922 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:44.657584    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:44.670157 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:44.670130    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:45.471546 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:45.471517    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg"
Jul 31 12:28:50.850384 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.850352    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"]
Jul 31 12:28:50.858372 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.858349    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:50.861078 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.861055    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 31 12:28:50.869779 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.869751    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"]
Jul 31 12:28:50.926991 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.926964    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hlf\" (UniqueName: \"kubernetes.io/projected/880003dc-67b7-41a8-9276-24cdc2822714-kube-api-access-26hlf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:50.927114 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.926997    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880003dc-67b7-41a8-9276-24cdc2822714-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:50.927114 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.927020    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:50.927114 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.927072    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:50.927239 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.927160    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:50.927239 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:50.927196    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.027955 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.027917    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.028132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.027970    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hlf\" (UniqueName: \"kubernetes.io/projected/880003dc-67b7-41a8-9276-24cdc2822714-kube-api-access-26hlf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.028132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.028000    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880003dc-67b7-41a8-9276-24cdc2822714-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.028132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.028033    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.028132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.028070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.028357 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.028144    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.028449 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.028425    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.028518 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.028461    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.028573 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.028538    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.030320 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.030298    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880003dc-67b7-41a8-9276-24cdc2822714-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.030599 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.030579    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880003dc-67b7-41a8-9276-24cdc2822714-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.035536 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.035514    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hlf\" (UniqueName: \"kubernetes.io/projected/880003dc-67b7-41a8-9276-24cdc2822714-kube-api-access-26hlf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bln7l\" (UID: \"880003dc-67b7-41a8-9276-24cdc2822714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.171410 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.171381    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:28:51.306203 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.306175    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"]
Jul 31 12:28:51.308593 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:28:51.308573    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880003dc_67b7_41a8_9276_24cdc2822714.slice/crio-71c356e74ffa1f9e4d92e36a1343e5ee94da778ed8e5c9c1e3726f588027aa1d WatchSource:0}: Error finding container 71c356e74ffa1f9e4d92e36a1343e5ee94da778ed8e5c9c1e3726f588027aa1d: Status 404 returned error can't find the container with id 71c356e74ffa1f9e4d92e36a1343e5ee94da778ed8e5c9c1e3726f588027aa1d
Jul 31 12:28:51.488517 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:51.488450    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l" event={"ID":"880003dc-67b7-41a8-9276-24cdc2822714","Type":"ContainerStarted","Data":"71c356e74ffa1f9e4d92e36a1343e5ee94da778ed8e5c9c1e3726f588027aa1d"}
Jul 31 12:28:52.494892 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:28:52.494857    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l" event={"ID":"880003dc-67b7-41a8-9276-24cdc2822714","Type":"ContainerStarted","Data":"052fe9194d8d5d0fb488e03b35b7a3e12fa37d02cb56f817b97753427d0f3e87"}
Jul 31 12:29:01.172358 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:01.172321    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:29:01.172753 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:01.172392    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:29:01.184942 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:01.184919    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:29:01.207427 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:01.207385    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l" podStartSLOduration=10.986898825 podStartE2EDuration="11.207370948s" totalImagesPullingTime="220.472123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:28:52.534364629 +0000 UTC m=+820.521586169" watchObservedRunningTime="2026-07-31 12:29:01.207370948 +0000 UTC m=+829.194592522"
Jul 31 12:29:01.545468 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:01.545394    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bln7l"
Jul 31 12:29:02.946758 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:02.946727    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"]
Jul 31 12:29:02.950639 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:02.950616    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:02.953182 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:02.953162    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 31 12:29:02.961291 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:02.961242    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"]
Jul 31 12:29:03.030539 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.030505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.030689 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.030557    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.030689 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.030669    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.030803 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.030732    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.030803 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.030765    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.030903 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.030827    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flmfb\" (UniqueName: \"kubernetes.io/projected/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-kube-api-access-flmfb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.131555 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.131522    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.131721 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.131566    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.131721 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.131584    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.131721 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.131611    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmfb\" (UniqueName: \"kubernetes.io/projected/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-kube-api-access-flmfb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.131721 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.131634    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.131721 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.131664    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.132120 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.132089    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.132120 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.132103    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.132299 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.132123    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.133804 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.133783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.133962 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.133947    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.138868 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.138852    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmfb\" (UniqueName: \"kubernetes.io/projected/ea7b93cf-7a1d-4b45-867a-2992bcd2fa51-kube-api-access-flmfb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fb6jd\" (UID: \"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.264665 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.264575    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:03.392150 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.392128    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"]
Jul 31 12:29:03.394189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:29:03.394160    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7b93cf_7a1d_4b45_867a_2992bcd2fa51.slice/crio-77d039f6f67266cc7847ed1295279ede0038cbf40a7c2465ccd714884d4c3a7c WatchSource:0}: Error finding container 77d039f6f67266cc7847ed1295279ede0038cbf40a7c2465ccd714884d4c3a7c: Status 404 returned error can't find the container with id 77d039f6f67266cc7847ed1295279ede0038cbf40a7c2465ccd714884d4c3a7c
Jul 31 12:29:03.544660 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:03.544569    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd" event={"ID":"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51","Type":"ContainerStarted","Data":"77d039f6f67266cc7847ed1295279ede0038cbf40a7c2465ccd714884d4c3a7c"}
Jul 31 12:29:04.550149 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:04.550111    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd" event={"ID":"ea7b93cf-7a1d-4b45-867a-2992bcd2fa51","Type":"ContainerStarted","Data":"1b98829e9442574f0d0e5e2a39be8ebdae311b5e11ed446c55edb17a17114210"}
Jul 31 12:29:04.580220 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:04.580165    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd" podStartSLOduration=2.320681557 podStartE2EDuration="2.580148994s" totalImagesPullingTime="259.467437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:29:04.578298596 +0000 UTC m=+832.565520138" watchObservedRunningTime="2026-07-31 12:29:04.580148994 +0000 UTC m=+832.567370535"
Jul 31 12:29:13.265648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:13.265588    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:13.266203 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:13.265684    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:13.278987 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:13.278962    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:29:13.604321 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:29:13.604225    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fb6jd"
Jul 31 12:30:00.142568 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:00.142536    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql"]
Jul 31 12:30:00.146173 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:00.146153    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql"
Jul 31 12:30:00.148759 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:00.148734    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lxtlm\""
Jul 31 12:30:00.153181 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:00.153158    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql"]
Jul 31 12:30:00.188108 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:00.188078    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql"
Jul 31 12:30:00.333578 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:00.333552    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql"]
Jul 31 12:30:00.336109 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:30:00.336078    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a4c00e_0358_4a0a_92ed_847414676cbb.slice/crio-8cbdffb206d22cf130ce102a3df8651aa7c6736d15b15232eed4438d14c76935 WatchSource:0}: Error finding container 8cbdffb206d22cf130ce102a3df8651aa7c6736d15b15232eed4438d14c76935: Status 404 returned error can't find the container with id 8cbdffb206d22cf130ce102a3df8651aa7c6736d15b15232eed4438d14c76935
Jul 31 12:30:00.802187 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:00.802155    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql" event={"ID":"85a4c00e-0358-4a0a-92ed-847414676cbb","Type":"ContainerStarted","Data":"8cbdffb206d22cf130ce102a3df8651aa7c6736d15b15232eed4438d14c76935"}
Jul 31 12:30:03.818799 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:03.818760    2586 generic.go:356] "Generic (PLEG): container finished" podID="85a4c00e-0358-4a0a-92ed-847414676cbb" containerID="aa342784df5370f574d1f903539bfc0ace5158f603289ea4e6b393353cfdbee7" exitCode=0
Jul 31 12:30:03.819156 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:03.818816    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql" event={"ID":"85a4c00e-0358-4a0a-92ed-847414676cbb","Type":"ContainerDied","Data":"aa342784df5370f574d1f903539bfc0ace5158f603289ea4e6b393353cfdbee7"}
Jul 31 12:30:04.955171 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:04.955148    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql"
Jul 31 12:30:05.828714 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:05.828679    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql" event={"ID":"85a4c00e-0358-4a0a-92ed-847414676cbb","Type":"ContainerDied","Data":"8cbdffb206d22cf130ce102a3df8651aa7c6736d15b15232eed4438d14c76935"}
Jul 31 12:30:05.828714 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:05.828718    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql"
Jul 31 12:30:05.829002 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:05.828729    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cbdffb206d22cf130ce102a3df8651aa7c6736d15b15232eed4438d14c76935"
Jul 31 12:30:17.632914 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.632874    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bcd57d65f-pqfnn"]
Jul 31 12:30:17.633519 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.633425    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a4c00e-0358-4a0a-92ed-847414676cbb" containerName="cleanup"
Jul 31 12:30:17.633519 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.633445    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a4c00e-0358-4a0a-92ed-847414676cbb" containerName="cleanup"
Jul 31 12:30:17.633651 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.633560    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a4c00e-0358-4a0a-92ed-847414676cbb" containerName="cleanup"
Jul 31 12:30:17.638696 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.638674    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcd57d65f-pqfnn"
Jul 31 12:30:17.656226 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.656183    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bcd57d65f-pqfnn"]
Jul 31 12:30:17.784991 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.784958    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b9bcd47f-94e1-45dd-81e1-36ce04c9c303-tls-cert\") pod \"authorino-bcd57d65f-pqfnn\" (UID: \"b9bcd47f-94e1-45dd-81e1-36ce04c9c303\") " pod="kuadrant-system/authorino-bcd57d65f-pqfnn"
Jul 31 12:30:17.785171 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.785009    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6qfk\" (UniqueName: \"kubernetes.io/projected/b9bcd47f-94e1-45dd-81e1-36ce04c9c303-kube-api-access-f6qfk\") pod \"authorino-bcd57d65f-pqfnn\" (UID: \"b9bcd47f-94e1-45dd-81e1-36ce04c9c303\") " pod="kuadrant-system/authorino-bcd57d65f-pqfnn"
Jul 31 12:30:17.885592 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.885508    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b9bcd47f-94e1-45dd-81e1-36ce04c9c303-tls-cert\") pod \"authorino-bcd57d65f-pqfnn\" (UID: \"b9bcd47f-94e1-45dd-81e1-36ce04c9c303\") " pod="kuadrant-system/authorino-bcd57d65f-pqfnn"
Jul 31 12:30:17.885592 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.885557    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6qfk\" (UniqueName: \"kubernetes.io/projected/b9bcd47f-94e1-45dd-81e1-36ce04c9c303-kube-api-access-f6qfk\") pod \"authorino-bcd57d65f-pqfnn\" (UID: \"b9bcd47f-94e1-45dd-81e1-36ce04c9c303\") " pod="kuadrant-system/authorino-bcd57d65f-pqfnn"
Jul 31 12:30:17.887978 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.887961    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b9bcd47f-94e1-45dd-81e1-36ce04c9c303-tls-cert\") pod \"authorino-bcd57d65f-pqfnn\" (UID: \"b9bcd47f-94e1-45dd-81e1-36ce04c9c303\") " pod="kuadrant-system/authorino-bcd57d65f-pqfnn"
Jul 31 12:30:17.901050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.901029    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6qfk\" (UniqueName: \"kubernetes.io/projected/b9bcd47f-94e1-45dd-81e1-36ce04c9c303-kube-api-access-f6qfk\") pod \"authorino-bcd57d65f-pqfnn\" (UID: \"b9bcd47f-94e1-45dd-81e1-36ce04c9c303\") " pod="kuadrant-system/authorino-bcd57d65f-pqfnn"
Jul 31 12:30:17.947571 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:17.947547    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcd57d65f-pqfnn"
Jul 31 12:30:18.093520 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:18.093488    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bcd57d65f-pqfnn"]
Jul 31 12:30:18.096920 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:30:18.096894    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9bcd47f_94e1_45dd_81e1_36ce04c9c303.slice/crio-24216b41f19faeb3a71ec833eea3df0a2a798370af805ae103f4afcdf21cb0e4 WatchSource:0}: Error finding container 24216b41f19faeb3a71ec833eea3df0a2a798370af805ae103f4afcdf21cb0e4: Status 404 returned error can't find the container with id 24216b41f19faeb3a71ec833eea3df0a2a798370af805ae103f4afcdf21cb0e4
Jul 31 12:30:18.888457 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:18.888424    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcd57d65f-pqfnn" event={"ID":"b9bcd47f-94e1-45dd-81e1-36ce04c9c303","Type":"ContainerStarted","Data":"24216b41f19faeb3a71ec833eea3df0a2a798370af805ae103f4afcdf21cb0e4"}
Jul 31 12:30:25.924791 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:25.924690    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcd57d65f-pqfnn" event={"ID":"b9bcd47f-94e1-45dd-81e1-36ce04c9c303","Type":"ContainerStarted","Data":"00db5119afe5e5ddfc0bb2d9b999396c9ebc50ee352a7ac53f4827c5be5324a6"}
Jul 31 12:30:25.947694 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:25.947641    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bcd57d65f-pqfnn" podStartSLOduration=1.381779971 podStartE2EDuration="8.947623315s" totalImagesPullingTime="7.565843344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:30:25.946411855 +0000 UTC m=+913.933633398" watchObservedRunningTime="2026-07-31 12:30:25.947623315 +0000 UTC m=+913.934844854"
Jul 31 12:30:25.982201 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:25.982162    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8bdcf4486-mdwjl"]
Jul 31 12:30:25.982443 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:25.982417    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8bdcf4486-mdwjl" podUID="e3792608-9414-4dd7-a04d-7952227bd887" containerName="authorino" containerID="cri-o://89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67" gracePeriod=30
Jul 31 12:30:26.227245 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.227217    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:30:26.256768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.256739    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3792608-9414-4dd7-a04d-7952227bd887-tls-cert\") pod \"e3792608-9414-4dd7-a04d-7952227bd887\" (UID: \"e3792608-9414-4dd7-a04d-7952227bd887\") "
Jul 31 12:30:26.256768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.256769    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9fl7\" (UniqueName: \"kubernetes.io/projected/e3792608-9414-4dd7-a04d-7952227bd887-kube-api-access-f9fl7\") pod \"e3792608-9414-4dd7-a04d-7952227bd887\" (UID: \"e3792608-9414-4dd7-a04d-7952227bd887\") "
Jul 31 12:30:26.258864 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.258830    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3792608-9414-4dd7-a04d-7952227bd887-kube-api-access-f9fl7" (OuterVolumeSpecName: "kube-api-access-f9fl7") pod "e3792608-9414-4dd7-a04d-7952227bd887" (UID: "e3792608-9414-4dd7-a04d-7952227bd887"). InnerVolumeSpecName "kube-api-access-f9fl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:30:26.267689 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.267659    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3792608-9414-4dd7-a04d-7952227bd887-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e3792608-9414-4dd7-a04d-7952227bd887" (UID: "e3792608-9414-4dd7-a04d-7952227bd887"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:30:26.357929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.357895    2586 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3792608-9414-4dd7-a04d-7952227bd887-tls-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:30:26.357929 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.357929    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9fl7\" (UniqueName: \"kubernetes.io/projected/e3792608-9414-4dd7-a04d-7952227bd887-kube-api-access-f9fl7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:30:26.931068 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.931029    2586 generic.go:356] "Generic (PLEG): container finished" podID="e3792608-9414-4dd7-a04d-7952227bd887" containerID="89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67" exitCode=0
Jul 31 12:30:26.931068 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.931069    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bdcf4486-mdwjl"
Jul 31 12:30:26.931597 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.931109    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bdcf4486-mdwjl" event={"ID":"e3792608-9414-4dd7-a04d-7952227bd887","Type":"ContainerDied","Data":"89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67"}
Jul 31 12:30:26.931597 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.931153    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bdcf4486-mdwjl" event={"ID":"e3792608-9414-4dd7-a04d-7952227bd887","Type":"ContainerDied","Data":"3df04f06ecbbde013d539e72f4f1aa387874e67cdac5de29d8bcf5844cc9c532"}
Jul 31 12:30:26.931597 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.931179    2586 scope.go:117] "RemoveContainer" containerID="89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67"
Jul 31 12:30:26.941522 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.941504    2586 scope.go:117] "RemoveContainer" containerID="89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67"
Jul 31 12:30:26.941797 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:30:26.941779    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67\": container with ID starting with 89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67 not found: ID does not exist" containerID="89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67"
Jul 31 12:30:26.941855 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.941805    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67"} err="failed to get container status \"89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67\": rpc error: code = NotFound desc = could not find container \"89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67\": container with ID starting with 89859e9c930b9eab60e4ac2dbbf19cfd789bb3736d4417a742f9444bcf4edf67 not found: ID does not exist"
Jul 31 12:30:26.960816 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.960785    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8bdcf4486-mdwjl"]
Jul 31 12:30:26.967704 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:26.967681    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8bdcf4486-mdwjl"]
Jul 31 12:30:28.484582 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:30:28.484554    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3792608-9414-4dd7-a04d-7952227bd887" path="/var/lib/kubelet/pods/e3792608-9414-4dd7-a04d-7952227bd887/volumes"
Jul 31 12:31:13.946237 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:13.946203    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-jz4m8"]
Jul 31 12:31:13.946651 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:13.946413    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c67bf5868-jz4m8" podUID="38c89a0f-fe55-4960-bd37-378ad36e0802" containerName="manager" containerID="cri-o://a5d3dc3dc0d7e32a4a24367807b46b65a7636ca42d6967c456e246293e7767c6" gracePeriod=30
Jul 31 12:31:14.160694 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:14.160659    2586 generic.go:356] "Generic (PLEG): container finished" podID="38c89a0f-fe55-4960-bd37-378ad36e0802" containerID="a5d3dc3dc0d7e32a4a24367807b46b65a7636ca42d6967c456e246293e7767c6" exitCode=0
Jul 31 12:31:14.160857 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:14.160711    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-jz4m8" event={"ID":"38c89a0f-fe55-4960-bd37-378ad36e0802","Type":"ContainerDied","Data":"a5d3dc3dc0d7e32a4a24367807b46b65a7636ca42d6967c456e246293e7767c6"}
Jul 31 12:31:14.200139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:14.200080    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:31:14.370839 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:14.370810    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svbbt\" (UniqueName: \"kubernetes.io/projected/38c89a0f-fe55-4960-bd37-378ad36e0802-kube-api-access-svbbt\") pod \"38c89a0f-fe55-4960-bd37-378ad36e0802\" (UID: \"38c89a0f-fe55-4960-bd37-378ad36e0802\") "
Jul 31 12:31:14.370987 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:14.370875    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38c89a0f-fe55-4960-bd37-378ad36e0802-webhook-cert\") pod \"38c89a0f-fe55-4960-bd37-378ad36e0802\" (UID: \"38c89a0f-fe55-4960-bd37-378ad36e0802\") "
Jul 31 12:31:14.372802 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:14.372772    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c89a0f-fe55-4960-bd37-378ad36e0802-kube-api-access-svbbt" (OuterVolumeSpecName: "kube-api-access-svbbt") pod "38c89a0f-fe55-4960-bd37-378ad36e0802" (UID: "38c89a0f-fe55-4960-bd37-378ad36e0802"). InnerVolumeSpecName "kube-api-access-svbbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:31:14.372915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:14.372859    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c89a0f-fe55-4960-bd37-378ad36e0802-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "38c89a0f-fe55-4960-bd37-378ad36e0802" (UID: "38c89a0f-fe55-4960-bd37-378ad36e0802"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:31:14.472080 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:14.472010    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svbbt\" (UniqueName: \"kubernetes.io/projected/38c89a0f-fe55-4960-bd37-378ad36e0802-kube-api-access-svbbt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:31:14.472080 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:14.472034    2586 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38c89a0f-fe55-4960-bd37-378ad36e0802-webhook-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:31:15.166231 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:15.166205    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-jz4m8"
Jul 31 12:31:15.166231 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:15.166221    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-jz4m8" event={"ID":"38c89a0f-fe55-4960-bd37-378ad36e0802","Type":"ContainerDied","Data":"6749630376b97d2ada8b5b03535d0469c6eb5e2961d7ac8f426d7f5fcfc35273"}
Jul 31 12:31:15.166678 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:15.166290    2586 scope.go:117] "RemoveContainer" containerID="a5d3dc3dc0d7e32a4a24367807b46b65a7636ca42d6967c456e246293e7767c6"
Jul 31 12:31:15.184591 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:15.184567    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-jz4m8"]
Jul 31 12:31:15.188794 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:15.188773    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-jz4m8"]
Jul 31 12:31:16.484723 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:31:16.484691    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38c89a0f-fe55-4960-bd37-378ad36e0802" path="/var/lib/kubelet/pods/38c89a0f-fe55-4960-bd37-378ad36e0802/volumes"
Jul 31 12:32:16.082441 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.082386    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c67bf5868-rchqj"]
Jul 31 12:32:16.083152 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.083130    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38c89a0f-fe55-4960-bd37-378ad36e0802" containerName="manager"
Jul 31 12:32:16.083230 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.083155    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="38c89a0f-fe55-4960-bd37-378ad36e0802" containerName="manager"
Jul 31 12:32:16.083230 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.083188    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3792608-9414-4dd7-a04d-7952227bd887" containerName="authorino"
Jul 31 12:32:16.083230 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.083197    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3792608-9414-4dd7-a04d-7952227bd887" containerName="authorino"
Jul 31 12:32:16.083474 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.083393    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3792608-9414-4dd7-a04d-7952227bd887" containerName="authorino"
Jul 31 12:32:16.083474 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.083419    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="38c89a0f-fe55-4960-bd37-378ad36e0802" containerName="manager"
Jul 31 12:32:16.087346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.087323    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:16.090676 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.090659    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-x7jpf\""
Jul 31 12:32:16.090676 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.090675    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 31 12:32:16.090816 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.090690    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 31 12:32:16.096573 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.096550    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-rchqj"]
Jul 31 12:32:16.186687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.186660    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5gs4\" (UniqueName: \"kubernetes.io/projected/80c174de-0d5d-4b1c-b134-db5c7f334066-kube-api-access-n5gs4\") pod \"maas-controller-7c67bf5868-rchqj\" (UID: \"80c174de-0d5d-4b1c-b134-db5c7f334066\") " pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:16.186830 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.186730    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80c174de-0d5d-4b1c-b134-db5c7f334066-webhook-cert\") pod \"maas-controller-7c67bf5868-rchqj\" (UID: \"80c174de-0d5d-4b1c-b134-db5c7f334066\") " pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:16.287221 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.287191    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80c174de-0d5d-4b1c-b134-db5c7f334066-webhook-cert\") pod \"maas-controller-7c67bf5868-rchqj\" (UID: \"80c174de-0d5d-4b1c-b134-db5c7f334066\") " pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:16.287390 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.287249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5gs4\" (UniqueName: \"kubernetes.io/projected/80c174de-0d5d-4b1c-b134-db5c7f334066-kube-api-access-n5gs4\") pod \"maas-controller-7c67bf5868-rchqj\" (UID: \"80c174de-0d5d-4b1c-b134-db5c7f334066\") " pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:16.289642 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.289619    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80c174de-0d5d-4b1c-b134-db5c7f334066-webhook-cert\") pod \"maas-controller-7c67bf5868-rchqj\" (UID: \"80c174de-0d5d-4b1c-b134-db5c7f334066\") " pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:16.295561 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.295527    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5gs4\" (UniqueName: \"kubernetes.io/projected/80c174de-0d5d-4b1c-b134-db5c7f334066-kube-api-access-n5gs4\") pod \"maas-controller-7c67bf5868-rchqj\" (UID: \"80c174de-0d5d-4b1c-b134-db5c7f334066\") " pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:16.398913 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.398879    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:16.530664 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.530634    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-rchqj"]
Jul 31 12:32:16.533149 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:32:16.533119    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c174de_0d5d_4b1c_b134_db5c7f334066.slice/crio-2bda86e7608f23e513bbcd6ae6bc95084d2267ea3d1aeaaa9991a3528df7684d WatchSource:0}: Error finding container 2bda86e7608f23e513bbcd6ae6bc95084d2267ea3d1aeaaa9991a3528df7684d: Status 404 returned error can't find the container with id 2bda86e7608f23e513bbcd6ae6bc95084d2267ea3d1aeaaa9991a3528df7684d
Jul 31 12:32:16.534377 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:16.534357    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 12:32:17.459204 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:17.459172    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-rchqj" event={"ID":"80c174de-0d5d-4b1c-b134-db5c7f334066","Type":"ContainerStarted","Data":"407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603"}
Jul 31 12:32:17.459602 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:17.459220    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-rchqj" event={"ID":"80c174de-0d5d-4b1c-b134-db5c7f334066","Type":"ContainerStarted","Data":"2bda86e7608f23e513bbcd6ae6bc95084d2267ea3d1aeaaa9991a3528df7684d"}
Jul 31 12:32:17.459602 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:17.459261    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:17.475713 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:17.475675    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c67bf5868-rchqj" podStartSLOduration=1.036539617 podStartE2EDuration="1.475661513s" totalImagesPullingTime="439.121896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:32:17.474850657 +0000 UTC m=+1025.462072198" watchObservedRunningTime="2026-07-31 12:32:17.475661513 +0000 UTC m=+1025.462883053"
Jul 31 12:32:28.466644 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:28.466607    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:33.132071 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.132031    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-rchqj"]
Jul 31 12:32:33.132474 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.132278    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c67bf5868-rchqj" podUID="80c174de-0d5d-4b1c-b134-db5c7f334066" containerName="manager" containerID="cri-o://407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603" gracePeriod=30
Jul 31 12:32:33.369687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.369660    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:33.432050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.432025    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5gs4\" (UniqueName: \"kubernetes.io/projected/80c174de-0d5d-4b1c-b134-db5c7f334066-kube-api-access-n5gs4\") pod \"80c174de-0d5d-4b1c-b134-db5c7f334066\" (UID: \"80c174de-0d5d-4b1c-b134-db5c7f334066\") "
Jul 31 12:32:33.432236 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.432089    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80c174de-0d5d-4b1c-b134-db5c7f334066-webhook-cert\") pod \"80c174de-0d5d-4b1c-b134-db5c7f334066\" (UID: \"80c174de-0d5d-4b1c-b134-db5c7f334066\") "
Jul 31 12:32:33.434146 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.434112    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c174de-0d5d-4b1c-b134-db5c7f334066-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "80c174de-0d5d-4b1c-b134-db5c7f334066" (UID: "80c174de-0d5d-4b1c-b134-db5c7f334066"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:32:33.434309 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.434207    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c174de-0d5d-4b1c-b134-db5c7f334066-kube-api-access-n5gs4" (OuterVolumeSpecName: "kube-api-access-n5gs4") pod "80c174de-0d5d-4b1c-b134-db5c7f334066" (UID: "80c174de-0d5d-4b1c-b134-db5c7f334066"). InnerVolumeSpecName "kube-api-access-n5gs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:32:33.530849 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.530815    2586 generic.go:356] "Generic (PLEG): container finished" podID="80c174de-0d5d-4b1c-b134-db5c7f334066" containerID="407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603" exitCode=0
Jul 31 12:32:33.531006 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.530855    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-rchqj"
Jul 31 12:32:33.531006 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.530896    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-rchqj" event={"ID":"80c174de-0d5d-4b1c-b134-db5c7f334066","Type":"ContainerDied","Data":"407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603"}
Jul 31 12:32:33.531006 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.530946    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-rchqj" event={"ID":"80c174de-0d5d-4b1c-b134-db5c7f334066","Type":"ContainerDied","Data":"2bda86e7608f23e513bbcd6ae6bc95084d2267ea3d1aeaaa9991a3528df7684d"}
Jul 31 12:32:33.531006 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.530968    2586 scope.go:117] "RemoveContainer" containerID="407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603"
Jul 31 12:32:33.532976 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.532951    2586 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80c174de-0d5d-4b1c-b134-db5c7f334066-webhook-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:32:33.532976 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.532970    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5gs4\" (UniqueName: \"kubernetes.io/projected/80c174de-0d5d-4b1c-b134-db5c7f334066-kube-api-access-n5gs4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:32:33.539744 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.539725    2586 scope.go:117] "RemoveContainer" containerID="407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603"
Jul 31 12:32:33.539971 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:32:33.539954    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603\": container with ID starting with 407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603 not found: ID does not exist" containerID="407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603"
Jul 31 12:32:33.540027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.539978    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603"} err="failed to get container status \"407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603\": rpc error: code = NotFound desc = could not find container \"407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603\": container with ID starting with 407ea1654ac3769e5082b822a7e39885a3ec5ea8540ec6343bae84fcb2e24603 not found: ID does not exist"
Jul 31 12:32:33.551779 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.551755    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-rchqj"]
Jul 31 12:32:33.558371 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:33.558351    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-rchqj"]
Jul 31 12:32:34.391116 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.391084    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c67bf5868-njj7d"]
Jul 31 12:32:34.391509 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.391460    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c174de-0d5d-4b1c-b134-db5c7f334066" containerName="manager"
Jul 31 12:32:34.391509 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.391472    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c174de-0d5d-4b1c-b134-db5c7f334066" containerName="manager"
Jul 31 12:32:34.391581 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.391536    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c174de-0d5d-4b1c-b134-db5c7f334066" containerName="manager"
Jul 31 12:32:34.394594 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.394574    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:32:34.397653 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.397630    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 31 12:32:34.397769 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.397752    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-x7jpf\""
Jul 31 12:32:34.397822 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.397769    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 31 12:32:34.405219 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.405198    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-njj7d"]
Jul 31 12:32:34.440881 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.440849    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e72d399-87d9-4a32-9c99-5e44b4616a22-webhook-cert\") pod \"maas-controller-7c67bf5868-njj7d\" (UID: \"0e72d399-87d9-4a32-9c99-5e44b4616a22\") " pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:32:34.441006 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.440885    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nbwr\" (UniqueName: \"kubernetes.io/projected/0e72d399-87d9-4a32-9c99-5e44b4616a22-kube-api-access-9nbwr\") pod \"maas-controller-7c67bf5868-njj7d\" (UID: \"0e72d399-87d9-4a32-9c99-5e44b4616a22\") " pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:32:34.484222 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.484197    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c174de-0d5d-4b1c-b134-db5c7f334066" path="/var/lib/kubelet/pods/80c174de-0d5d-4b1c-b134-db5c7f334066/volumes"
Jul 31 12:32:34.541545 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.541515    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e72d399-87d9-4a32-9c99-5e44b4616a22-webhook-cert\") pod \"maas-controller-7c67bf5868-njj7d\" (UID: \"0e72d399-87d9-4a32-9c99-5e44b4616a22\") " pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:32:34.541688 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.541553    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbwr\" (UniqueName: \"kubernetes.io/projected/0e72d399-87d9-4a32-9c99-5e44b4616a22-kube-api-access-9nbwr\") pod \"maas-controller-7c67bf5868-njj7d\" (UID: \"0e72d399-87d9-4a32-9c99-5e44b4616a22\") " pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:32:34.543870 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.543846    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e72d399-87d9-4a32-9c99-5e44b4616a22-webhook-cert\") pod \"maas-controller-7c67bf5868-njj7d\" (UID: \"0e72d399-87d9-4a32-9c99-5e44b4616a22\") " pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:32:34.549548 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.549523    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbwr\" (UniqueName: \"kubernetes.io/projected/0e72d399-87d9-4a32-9c99-5e44b4616a22-kube-api-access-9nbwr\") pod \"maas-controller-7c67bf5868-njj7d\" (UID: \"0e72d399-87d9-4a32-9c99-5e44b4616a22\") " pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:32:34.704839 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.704818    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:32:34.832383 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:34.832360    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c67bf5868-njj7d"]
Jul 31 12:32:34.834535 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:32:34.834511    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e72d399_87d9_4a32_9c99_5e44b4616a22.slice/crio-8a85c6009fed81d20c03fc35022bfcf82407bbb2fc9c81339118252ac5eed427 WatchSource:0}: Error finding container 8a85c6009fed81d20c03fc35022bfcf82407bbb2fc9c81339118252ac5eed427: Status 404 returned error can't find the container with id 8a85c6009fed81d20c03fc35022bfcf82407bbb2fc9c81339118252ac5eed427
Jul 31 12:32:35.544436 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:35.544400    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-njj7d" event={"ID":"0e72d399-87d9-4a32-9c99-5e44b4616a22","Type":"ContainerStarted","Data":"cfe5a4fe7b1377496dde3a5041278b5be950f78620315f60e5d9eb6ed97ef258"}
Jul 31 12:32:35.544887 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:35.544462    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c67bf5868-njj7d" event={"ID":"0e72d399-87d9-4a32-9c99-5e44b4616a22","Type":"ContainerStarted","Data":"8a85c6009fed81d20c03fc35022bfcf82407bbb2fc9c81339118252ac5eed427"}
Jul 31 12:32:35.544887 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:35.544490    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:32:35.561704 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:35.561600    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c67bf5868-njj7d" podStartSLOduration=1.091777473 podStartE2EDuration="1.561582439s" totalImagesPullingTime="469.804966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:32:35.560598094 +0000 UTC m=+1043.547819637" watchObservedRunningTime="2026-07-31 12:32:35.561582439 +0000 UTC m=+1043.548803984"
Jul 31 12:32:46.550781 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:32:46.550746    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c67bf5868-njj7d"
Jul 31 12:42:57.089095 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.089062    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"]
Jul 31 12:42:57.092228 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.092209    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.097118 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.097094    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6d50556f-kserve-self-signed-certs\""
Jul 31 12:42:57.121936 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.121918    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"]
Jul 31 12:42:57.265785 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.265751    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-home\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.265785 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.265789    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136a0b98-9469-44e2-9f4a-40ac3b160977-tls-certs\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.266103 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.265819    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-model-cache\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.266103 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.265944    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-dshm\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.266103 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.265983    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-tmp-dir\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.266103 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.266041    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94x9\" (UniqueName: \"kubernetes.io/projected/136a0b98-9469-44e2-9f4a-40ac3b160977-kube-api-access-b94x9\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.367309 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.367197    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-home\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.367309 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.367240    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136a0b98-9469-44e2-9f4a-40ac3b160977-tls-certs\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.367309 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.367290    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-model-cache\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.367576 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.367362    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-dshm\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.367576 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.367383    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-tmp-dir\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.367576 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.367432    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94x9\" (UniqueName: \"kubernetes.io/projected/136a0b98-9469-44e2-9f4a-40ac3b160977-kube-api-access-b94x9\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.375067 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.368306    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-tmp-dir\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.375067 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.368097    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-model-cache\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.375067 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.370572    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-home\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.375067 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.370713    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-dshm\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.375067 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.374591    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136a0b98-9469-44e2-9f4a-40ac3b160977-tls-certs\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.376966 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.376941    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94x9\" (UniqueName: \"kubernetes.io/projected/136a0b98-9469-44e2-9f4a-40ac3b160977-kube-api-access-b94x9\") pod \"e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") " pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.400528 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.400507    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:42:57.527995 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:57.527967    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"]
Jul 31 12:42:57.529766 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:42:57.529741    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod136a0b98_9469_44e2_9f4a_40ac3b160977.slice/crio-3785fa5f4d0c52e0f3ea68700f5cf7b4a364a4cfe5ddaa230b07e37b6b9c5c20 WatchSource:0}: Error finding container 3785fa5f4d0c52e0f3ea68700f5cf7b4a364a4cfe5ddaa230b07e37b6b9c5c20: Status 404 returned error can't find the container with id 3785fa5f4d0c52e0f3ea68700f5cf7b4a364a4cfe5ddaa230b07e37b6b9c5c20
Jul 31 12:42:58.424055 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:58.424021    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442" event={"ID":"136a0b98-9469-44e2-9f4a-40ac3b160977","Type":"ContainerStarted","Data":"25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4"}
Jul 31 12:42:58.424055 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:58.424059    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442" event={"ID":"136a0b98-9469-44e2-9f4a-40ac3b160977","Type":"ContainerStarted","Data":"3785fa5f4d0c52e0f3ea68700f5cf7b4a364a4cfe5ddaa230b07e37b6b9c5c20"}
Jul 31 12:42:58.441397 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:42:58.441350    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442" podStartSLOduration=1.441334364 podStartE2EDuration="1.441334364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:42:58.439720077 +0000 UTC m=+1666.426941617" watchObservedRunningTime="2026-07-31 12:42:58.441334364 +0000 UTC m=+1666.428555917"
Jul 31 12:43:07.401241 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:07.401209    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:43:07.401741 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:07.401496    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:43:07.413941 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:07.413918    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:43:07.481210 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:07.481190    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:43:29.082221 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:29.082189    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"]
Jul 31 12:43:44.101056 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.100968    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442" podUID="136a0b98-9469-44e2-9f4a-40ac3b160977" containerName="main" containerID="cri-o://25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4" gracePeriod=45
Jul 31 12:43:44.346531 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.346509    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:43:44.380169 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380113    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-dshm\") pod \"136a0b98-9469-44e2-9f4a-40ac3b160977\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") "
Jul 31 12:43:44.380169 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380146    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-model-cache\") pod \"136a0b98-9469-44e2-9f4a-40ac3b160977\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") "
Jul 31 12:43:44.380379 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380175    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136a0b98-9469-44e2-9f4a-40ac3b160977-tls-certs\") pod \"136a0b98-9469-44e2-9f4a-40ac3b160977\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") "
Jul 31 12:43:44.380379 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380191    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b94x9\" (UniqueName: \"kubernetes.io/projected/136a0b98-9469-44e2-9f4a-40ac3b160977-kube-api-access-b94x9\") pod \"136a0b98-9469-44e2-9f4a-40ac3b160977\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") "
Jul 31 12:43:44.380379 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380342    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-home\") pod \"136a0b98-9469-44e2-9f4a-40ac3b160977\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") "
Jul 31 12:43:44.380537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380437    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-model-cache" (OuterVolumeSpecName: "model-cache") pod "136a0b98-9469-44e2-9f4a-40ac3b160977" (UID: "136a0b98-9469-44e2-9f4a-40ac3b160977"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:43:44.380537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380446    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-tmp-dir\") pod \"136a0b98-9469-44e2-9f4a-40ac3b160977\" (UID: \"136a0b98-9469-44e2-9f4a-40ac3b160977\") "
Jul 31 12:43:44.380680 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380632    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-home" (OuterVolumeSpecName: "home") pod "136a0b98-9469-44e2-9f4a-40ac3b160977" (UID: "136a0b98-9469-44e2-9f4a-40ac3b160977"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:43:44.380741 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380697    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "136a0b98-9469-44e2-9f4a-40ac3b160977" (UID: "136a0b98-9469-44e2-9f4a-40ac3b160977"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:43:44.381052 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380858    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:43:44.381052 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380877    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:43:44.381052 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.380891    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:43:44.382441 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.382413    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136a0b98-9469-44e2-9f4a-40ac3b160977-kube-api-access-b94x9" (OuterVolumeSpecName: "kube-api-access-b94x9") pod "136a0b98-9469-44e2-9f4a-40ac3b160977" (UID: "136a0b98-9469-44e2-9f4a-40ac3b160977"). InnerVolumeSpecName "kube-api-access-b94x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:43:44.382553 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.382504    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-dshm" (OuterVolumeSpecName: "dshm") pod "136a0b98-9469-44e2-9f4a-40ac3b160977" (UID: "136a0b98-9469-44e2-9f4a-40ac3b160977"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:43:44.382553 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.382542    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136a0b98-9469-44e2-9f4a-40ac3b160977-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "136a0b98-9469-44e2-9f4a-40ac3b160977" (UID: "136a0b98-9469-44e2-9f4a-40ac3b160977"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:43:44.481216 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.481188    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a0b98-9469-44e2-9f4a-40ac3b160977-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:43:44.481216 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.481211    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136a0b98-9469-44e2-9f4a-40ac3b160977-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:43:44.481412 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.481225    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b94x9\" (UniqueName: \"kubernetes.io/projected/136a0b98-9469-44e2-9f4a-40ac3b160977-kube-api-access-b94x9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:43:44.646167 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.646092    2586 generic.go:356] "Generic (PLEG): container finished" podID="136a0b98-9469-44e2-9f4a-40ac3b160977" containerID="25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4" exitCode=0
Jul 31 12:43:44.646167 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.646139    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"
Jul 31 12:43:44.646363 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.646178    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442" event={"ID":"136a0b98-9469-44e2-9f4a-40ac3b160977","Type":"ContainerDied","Data":"25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4"}
Jul 31 12:43:44.646363 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.646220    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442" event={"ID":"136a0b98-9469-44e2-9f4a-40ac3b160977","Type":"ContainerDied","Data":"3785fa5f4d0c52e0f3ea68700f5cf7b4a364a4cfe5ddaa230b07e37b6b9c5c20"}
Jul 31 12:43:44.646363 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.646236    2586 scope.go:117] "RemoveContainer" containerID="25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4"
Jul 31 12:43:44.654481 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.654462    2586 scope.go:117] "RemoveContainer" containerID="25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4"
Jul 31 12:43:44.654725 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:43:44.654711    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4\": container with ID starting with 25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4 not found: ID does not exist" containerID="25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4"
Jul 31 12:43:44.654771 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.654732    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4"} err="failed to get container status \"25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4\": rpc error: code = NotFound desc = could not find container \"25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4\": container with ID starting with 25f57ed672e1d138ea38153c6a3af0591ba12f3f4ec6c31f3da78b8aca4acfc4 not found: ID does not exist"
Jul 31 12:43:44.661661 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.661642    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"]
Jul 31 12:43:44.665565 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:44.665542    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6d50556f-kserve-5588c9547f-dp442"]
Jul 31 12:43:46.484914 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:43:46.484881    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="136a0b98-9469-44e2-9f4a-40ac3b160977" path="/var/lib/kubelet/pods/136a0b98-9469-44e2-9f4a-40ac3b160977/volumes"
Jul 31 12:45:00.138711 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:00.138674    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh"]
Jul 31 12:45:00.139210 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:00.139195    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136a0b98-9469-44e2-9f4a-40ac3b160977" containerName="main"
Jul 31 12:45:00.139254 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:00.139213    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="136a0b98-9469-44e2-9f4a-40ac3b160977" containerName="main"
Jul 31 12:45:00.139343 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:00.139330    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="136a0b98-9469-44e2-9f4a-40ac3b160977" containerName="main"
Jul 31 12:45:00.142840 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:00.142823    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh"
Jul 31 12:45:00.156286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:00.151639    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lxtlm\""
Jul 31 12:45:00.162826 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:00.162796    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh"]
Jul 31 12:45:00.163202 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:00.163178    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh"
Jul 31 12:45:00.512210 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:00.512184    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh"]
Jul 31 12:45:00.514723 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:45:00.514691    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560ddcb9_4b7c_4459_a03c_d3e28cad40f6.slice/crio-3cd6c2564c0d65fc2b5b03678db8bab1147e7391da758f62f91e8c8c1e846fef WatchSource:0}: Error finding container 3cd6c2564c0d65fc2b5b03678db8bab1147e7391da758f62f91e8c8c1e846fef: Status 404 returned error can't find the container with id 3cd6c2564c0d65fc2b5b03678db8bab1147e7391da758f62f91e8c8c1e846fef
Jul 31 12:45:01.012024 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:01.011988    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh" event={"ID":"560ddcb9-4b7c-4459-a03c-d3e28cad40f6","Type":"ContainerStarted","Data":"ce2894a5b18b04c7bb965dd0daed409e6d0edc32d4915b4ccc159de475917a86"}
Jul 31 12:45:01.012173 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:01.012043    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh" event={"ID":"560ddcb9-4b7c-4459-a03c-d3e28cad40f6","Type":"ContainerStarted","Data":"3cd6c2564c0d65fc2b5b03678db8bab1147e7391da758f62f91e8c8c1e846fef"}
Jul 31 12:45:01.033144 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:01.033104    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh" podStartSLOduration=1.033088534 podStartE2EDuration="1.033088534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:45:01.031197183 +0000 UTC m=+1789.018418723" watchObservedRunningTime="2026-07-31 12:45:01.033088534 +0000 UTC m=+1789.020310073"
Jul 31 12:45:02.018132 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:02.018098    2586 generic.go:356] "Generic (PLEG): container finished" podID="560ddcb9-4b7c-4459-a03c-d3e28cad40f6" containerID="ce2894a5b18b04c7bb965dd0daed409e6d0edc32d4915b4ccc159de475917a86" exitCode=0
Jul 31 12:45:02.018528 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:02.018190    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh" event={"ID":"560ddcb9-4b7c-4459-a03c-d3e28cad40f6","Type":"ContainerDied","Data":"ce2894a5b18b04c7bb965dd0daed409e6d0edc32d4915b4ccc159de475917a86"}
Jul 31 12:45:03.152577 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:03.152545    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh"
Jul 31 12:45:04.028708 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:04.028670    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh" event={"ID":"560ddcb9-4b7c-4459-a03c-d3e28cad40f6","Type":"ContainerDied","Data":"3cd6c2564c0d65fc2b5b03678db8bab1147e7391da758f62f91e8c8c1e846fef"}
Jul 31 12:45:04.028872 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:04.028718    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd6c2564c0d65fc2b5b03678db8bab1147e7391da758f62f91e8c8c1e846fef"
Jul 31 12:45:04.028872 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:45:04.028684    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758365-jhkbh"
Jul 31 12:47:24.335856 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.335824    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"]
Jul 31 12:47:24.336475 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.336456    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560ddcb9-4b7c-4459-a03c-d3e28cad40f6" containerName="cleanup"
Jul 31 12:47:24.336564 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.336477    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="560ddcb9-4b7c-4459-a03c-d3e28cad40f6" containerName="cleanup"
Jul 31 12:47:24.336620 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.336576    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="560ddcb9-4b7c-4459-a03c-d3e28cad40f6" containerName="cleanup"
Jul 31 12:47:24.339498 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.339478    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.341859 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.341835    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-84c24c8b-openshift-default-dockercfg-dtg6l\""
Jul 31 12:47:24.361178 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.361154    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"]
Jul 31 12:47:24.474847 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.474808    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istiod-ca-cert\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.474847 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.474845    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-podinfo\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.475089 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.474866    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-data\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.475089 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.474958    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcrbr\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-kube-api-access-xcrbr\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.475089 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.474999    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-certs\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.475089 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.475031    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-token\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.475089 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.475062    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-credential-socket\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.475371 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.475095    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-envoy\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.475371 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.475122    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-socket\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.576603 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.576571    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-podinfo\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.576603 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.576605    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-data\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.576819 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.576640    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcrbr\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-kube-api-access-xcrbr\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.576819 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.576667    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-certs\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.576819 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.576695    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-token\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.576819 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.576721    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-credential-socket\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.576819 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.576749    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-envoy\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.576819 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.576773    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-socket\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.577124 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.576872    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istiod-ca-cert\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.577124 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.577095    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-data\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.577230 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.577151    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-certs\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.577230 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.577198    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-socket\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.577452 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.577431    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-credential-socket\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.577813 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.577790    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istiod-ca-cert\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.579214 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.579191    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-envoy\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.579327 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.579278    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-podinfo\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.584377 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.584354    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcrbr\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-kube-api-access-xcrbr\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.584484 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.584467    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-token\") pod \"e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") " pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.649509 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.649481    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:24.992052 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.992005    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"]
Jul 31 12:47:24.994314 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:47:24.994282    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd37787f_4bd7_4f21_b4d0_3337cb79faa8.slice/crio-0a7079fd9fcff2652c4670cdf93e1737e09dc5002015098c0ed64ed5e0a05e14 WatchSource:0}: Error finding container 0a7079fd9fcff2652c4670cdf93e1737e09dc5002015098c0ed64ed5e0a05e14: Status 404 returned error can't find the container with id 0a7079fd9fcff2652c4670cdf93e1737e09dc5002015098c0ed64ed5e0a05e14
Jul 31 12:47:24.996568 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.996525    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:47:24.996668 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.996597    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:47:24.996668 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:24.996627    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:47:25.714300 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:25.714253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" event={"ID":"cd37787f-4bd7-4f21-b4d0-3337cb79faa8","Type":"ContainerStarted","Data":"8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476"}
Jul 31 12:47:25.714679 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:25.714319    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" event={"ID":"cd37787f-4bd7-4f21-b4d0-3337cb79faa8","Type":"ContainerStarted","Data":"0a7079fd9fcff2652c4670cdf93e1737e09dc5002015098c0ed64ed5e0a05e14"}
Jul 31 12:47:25.739188 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:25.739136    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" podStartSLOduration=1.739120352 podStartE2EDuration="1.739120352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:47:25.737054129 +0000 UTC m=+1933.724275666" watchObservedRunningTime="2026-07-31 12:47:25.739120352 +0000 UTC m=+1933.726341892"
Jul 31 12:47:26.650233 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:26.650195    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:26.655070 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:26.655047    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:26.719026 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:26.718991    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:26.719988 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:26.719969    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:28.169145 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.169107    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"]
Jul 31 12:47:28.173111 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.173091    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.175721 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.175698    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-84c24c8b-serving-cert\""
Jul 31 12:47:28.180152 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.180128    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"]
Jul 31 12:47:28.309905 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.309872    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvcj\" (UniqueName: \"kubernetes.io/projected/46b87365-0cd5-4a26-9955-8ff168e70adb-kube-api-access-zpvcj\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.310065 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.309942    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b87365-0cd5-4a26-9955-8ff168e70adb-openshift-service-ca\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.310065 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.309960    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b87365-0cd5-4a26-9955-8ff168e70adb-maas-api-tls\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.411316 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.411259    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b87365-0cd5-4a26-9955-8ff168e70adb-maas-api-tls\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.411466 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.411374    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvcj\" (UniqueName: \"kubernetes.io/projected/46b87365-0cd5-4a26-9955-8ff168e70adb-kube-api-access-zpvcj\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.411466 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:47:28.411410    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-serving-cert: secret "maas-api-e2e-ait-84c24c8b-serving-cert" not found
Jul 31 12:47:28.411466 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.411440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b87365-0cd5-4a26-9955-8ff168e70adb-openshift-service-ca\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.411571 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:47:28.411469    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b87365-0cd5-4a26-9955-8ff168e70adb-maas-api-tls podName:46b87365-0cd5-4a26-9955-8ff168e70adb nodeName:}" failed. No retries permitted until 2026-07-31 12:47:28.911452224 +0000 UTC m=+1936.898673742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46b87365-0cd5-4a26-9955-8ff168e70adb-maas-api-tls") pod "maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2" (UID: "46b87365-0cd5-4a26-9955-8ff168e70adb") : secret "maas-api-e2e-ait-84c24c8b-serving-cert" not found
Jul 31 12:47:28.412024 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.412006    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b87365-0cd5-4a26-9955-8ff168e70adb-openshift-service-ca\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.429028 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.428961    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvcj\" (UniqueName: \"kubernetes.io/projected/46b87365-0cd5-4a26-9955-8ff168e70adb-kube-api-access-zpvcj\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.915859 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.915830    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b87365-0cd5-4a26-9955-8ff168e70adb-maas-api-tls\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.918209 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.918187    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b87365-0cd5-4a26-9955-8ff168e70adb-maas-api-tls\") pod \"maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:28.934963 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.934938    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"]
Jul 31 12:47:28.938582 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.938566    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:28.941394 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.941376    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-84c24c8b\""
Jul 31 12:47:28.941497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.941399    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-84c24c8b-dockercfg-fmk72\""
Jul 31 12:47:28.952988 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.952962    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"]
Jul 31 12:47:28.967051 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.967028    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"]
Jul 31 12:47:28.970409 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.970393    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:28.982729 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:28.982709    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"]
Jul 31 12:47:29.016960 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.016933    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcaf387-d75b-4b76-b4f2-e2a721c129f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw\" (UID: \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:29.017093 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.016995    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw48d\" (UniqueName: \"kubernetes.io/projected/abcaf387-d75b-4b76-b4f2-e2a721c129f6-kube-api-access-lw48d\") pod \"payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw\" (UID: \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:29.086742 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.086715    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:29.118114 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.118083    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcaf387-d75b-4b76-b4f2-e2a721c129f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw\" (UID: \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:29.118250 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.118138    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz57q\" (UniqueName: \"kubernetes.io/projected/8bf370ba-e510-4fb6-8860-2372a1e53b2d-kube-api-access-vz57q\") pod \"payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j\" (UID: \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:29.118250 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.118233    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf370ba-e510-4fb6-8860-2372a1e53b2d-plugins-config-volume\") pod \"payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j\" (UID: \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:29.118343 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.118253    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw48d\" (UniqueName: \"kubernetes.io/projected/abcaf387-d75b-4b76-b4f2-e2a721c129f6-kube-api-access-lw48d\") pod \"payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw\" (UID: \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:29.119073 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.119048    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcaf387-d75b-4b76-b4f2-e2a721c129f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw\" (UID: \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:29.127179 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.127153    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw48d\" (UniqueName: \"kubernetes.io/projected/abcaf387-d75b-4b76-b4f2-e2a721c129f6-kube-api-access-lw48d\") pod \"payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw\" (UID: \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:29.219242 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.219210    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz57q\" (UniqueName: \"kubernetes.io/projected/8bf370ba-e510-4fb6-8860-2372a1e53b2d-kube-api-access-vz57q\") pod \"payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j\" (UID: \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:29.219676 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.219249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf370ba-e510-4fb6-8860-2372a1e53b2d-plugins-config-volume\") pod \"payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j\" (UID: \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:29.219914 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.219896    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf370ba-e510-4fb6-8860-2372a1e53b2d-plugins-config-volume\") pod \"payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j\" (UID: \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:29.232889 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.232864    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz57q\" (UniqueName: \"kubernetes.io/projected/8bf370ba-e510-4fb6-8860-2372a1e53b2d-kube-api-access-vz57q\") pod \"payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j\" (UID: \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:29.247348 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.247326    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"]
Jul 31 12:47:29.248615 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:47:29.248588    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b87365_0cd5_4a26_9955_8ff168e70adb.slice/crio-4ecd918774bd9caa81b69c766549ffbe35c11ed5e3058cd474c48d47d86c9fbc WatchSource:0}: Error finding container 4ecd918774bd9caa81b69c766549ffbe35c11ed5e3058cd474c48d47d86c9fbc: Status 404 returned error can't find the container with id 4ecd918774bd9caa81b69c766549ffbe35c11ed5e3058cd474c48d47d86c9fbc
Jul 31 12:47:29.249571 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.249554    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:29.249979 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.249965    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 12:47:29.282327 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.282308    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:29.388525 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.388501    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"]
Jul 31 12:47:29.390228 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:47:29.390200    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcaf387_d75b_4b76_b4f2_e2a721c129f6.slice/crio-ffd03987892d669a3bd530c67d582d271997468c6cb1fe8dc00ca0ee23675907 WatchSource:0}: Error finding container ffd03987892d669a3bd530c67d582d271997468c6cb1fe8dc00ca0ee23675907: Status 404 returned error can't find the container with id ffd03987892d669a3bd530c67d582d271997468c6cb1fe8dc00ca0ee23675907
Jul 31 12:47:29.636444 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.636422    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"]
Jul 31 12:47:29.638999 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:47:29.638971    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf370ba_e510_4fb6_8860_2372a1e53b2d.slice/crio-c89e024e2dcc81b3c572b8f8e3b7f48f0322070ff4605e44aecb561184add3d4 WatchSource:0}: Error finding container c89e024e2dcc81b3c572b8f8e3b7f48f0322070ff4605e44aecb561184add3d4: Status 404 returned error can't find the container with id c89e024e2dcc81b3c572b8f8e3b7f48f0322070ff4605e44aecb561184add3d4
Jul 31 12:47:29.735595 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.735558    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw" event={"ID":"abcaf387-d75b-4b76-b4f2-e2a721c129f6","Type":"ContainerStarted","Data":"9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023"}
Jul 31 12:47:29.735801 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.735624    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw" event={"ID":"abcaf387-d75b-4b76-b4f2-e2a721c129f6","Type":"ContainerStarted","Data":"ffd03987892d669a3bd530c67d582d271997468c6cb1fe8dc00ca0ee23675907"}
Jul 31 12:47:29.735801 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.735735    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:29.736798 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.736766    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2" event={"ID":"46b87365-0cd5-4a26-9955-8ff168e70adb","Type":"ContainerStarted","Data":"4ecd918774bd9caa81b69c766549ffbe35c11ed5e3058cd474c48d47d86c9fbc"}
Jul 31 12:47:29.738585 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.738562    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j" event={"ID":"8bf370ba-e510-4fb6-8860-2372a1e53b2d","Type":"ContainerStarted","Data":"294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f"}
Jul 31 12:47:29.738700 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.738607    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j" event={"ID":"8bf370ba-e510-4fb6-8860-2372a1e53b2d","Type":"ContainerStarted","Data":"c89e024e2dcc81b3c572b8f8e3b7f48f0322070ff4605e44aecb561184add3d4"}
Jul 31 12:47:29.738762 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.738711    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:29.751591 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.751548    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw" podStartSLOduration=1.751533732 podStartE2EDuration="1.751533732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:47:29.749853258 +0000 UTC m=+1937.737074812" watchObservedRunningTime="2026-07-31 12:47:29.751533732 +0000 UTC m=+1937.738755271"
Jul 31 12:47:29.766128 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:29.766088    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j" podStartSLOduration=1.76607399 podStartE2EDuration="1.76607399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:47:29.764668404 +0000 UTC m=+1937.751889943" watchObservedRunningTime="2026-07-31 12:47:29.76607399 +0000 UTC m=+1937.753295529"
Jul 31 12:47:30.744380 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:30.744253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2" event={"ID":"46b87365-0cd5-4a26-9955-8ff168e70adb","Type":"ContainerStarted","Data":"ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12"}
Jul 31 12:47:30.764168 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:30.764122    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2" podStartSLOduration=2.267662558 podStartE2EDuration="2.764107349s" totalImagesPullingTime="496.444791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:47:30.762309066 +0000 UTC m=+1938.749530608" watchObservedRunningTime="2026-07-31 12:47:30.764107349 +0000 UTC m=+1938.751328919"
Jul 31 12:47:31.752865 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:31.752803    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:37.767071 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:37.767042    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:47:40.746114 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:40.746080    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:47:40.746530 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:40.746238    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:47:40.801845 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:40.801817    2586 generic.go:356] "Generic (PLEG): container finished" podID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerID="8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476" exitCode=0
Jul 31 12:47:40.801990 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:40.801841    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" event={"ID":"cd37787f-4bd7-4f21-b4d0-3337cb79faa8","Type":"ContainerDied","Data":"8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476"}
Jul 31 12:47:40.802249 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:40.802233    2586 scope.go:117] "RemoveContainer" containerID="8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476"
Jul 31 12:47:40.803340 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:40.803308    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:47:40.803440 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:40.803392    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:47:40.803440 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:40.803429    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:47:41.719878 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:41.719847    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:41.808055 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:41.808018    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" event={"ID":"cd37787f-4bd7-4f21-b4d0-3337cb79faa8","Type":"ContainerStarted","Data":"5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f"}
Jul 31 12:47:42.652328 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:42.652294    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:42.652680 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:42.652654    2586 patch_prober.go:28] interesting pod/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 31 12:47:42.652854 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:42.652830    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 31 12:47:43.650660 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:43.650622    2586 patch_prober.go:28] interesting pod/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 31 12:47:43.651112 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:43.650691    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 31 12:47:44.650125 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:44.650080    2586 patch_prober.go:28] interesting pod/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 31 12:47:44.650345 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:44.650147    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 31 12:47:45.650615 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:45.650574    2586 patch_prober.go:28] interesting pod/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 31 12:47:45.651214 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:45.651175    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 31 12:47:46.650701 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:46.650653    2586 patch_prober.go:28] interesting pod/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 31 12:47:46.651198 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:46.650728    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 31 12:47:47.653751 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:47.653723    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:47.654175 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:47.654034    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:47:47.654797 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:47:47.654779    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:48:08.838809 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:08.838619    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"]
Jul 31 12:48:08.839632 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:08.839601    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2" podUID="46b87365-0cd5-4a26-9955-8ff168e70adb" containerName="maas-api" containerID="cri-o://ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12" gracePeriod=30
Jul 31 12:48:08.891433 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:08.891401    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"]
Jul 31 12:48:08.891876 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:08.891825    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j" podUID="8bf370ba-e510-4fb6-8860-2372a1e53b2d" containerName="payload-processing" containerID="cri-o://294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f" gracePeriod=30
Jul 31 12:48:08.894825 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:08.894798    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"]
Jul 31 12:48:08.895019 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:08.894994    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw" podUID="abcaf387-d75b-4b76-b4f2-e2a721c129f6" containerName="payload-pre-processing" containerID="cri-o://9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023" gracePeriod=30
Jul 31 12:48:09.161565 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.161003    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"]
Jul 31 12:48:09.161565 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.161249    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy" containerID="cri-o://5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f" gracePeriod=30
Jul 31 12:48:09.165172 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.165036    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:48:09.189331 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.189208    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:48:09.217155 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.217119    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:48:09.301847 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.301814    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf370ba-e510-4fb6-8860-2372a1e53b2d-plugins-config-volume\") pod \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\" (UID: \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\") "
Jul 31 12:48:09.302027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.301865    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b87365-0cd5-4a26-9955-8ff168e70adb-maas-api-tls\") pod \"46b87365-0cd5-4a26-9955-8ff168e70adb\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") "
Jul 31 12:48:09.302027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.301890    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpvcj\" (UniqueName: \"kubernetes.io/projected/46b87365-0cd5-4a26-9955-8ff168e70adb-kube-api-access-zpvcj\") pod \"46b87365-0cd5-4a26-9955-8ff168e70adb\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") "
Jul 31 12:48:09.302027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.301935    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcaf387-d75b-4b76-b4f2-e2a721c129f6-plugins-config-volume\") pod \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\" (UID: \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\") "
Jul 31 12:48:09.302027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.301984    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b87365-0cd5-4a26-9955-8ff168e70adb-openshift-service-ca\") pod \"46b87365-0cd5-4a26-9955-8ff168e70adb\" (UID: \"46b87365-0cd5-4a26-9955-8ff168e70adb\") "
Jul 31 12:48:09.302027 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.302020    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz57q\" (UniqueName: \"kubernetes.io/projected/8bf370ba-e510-4fb6-8860-2372a1e53b2d-kube-api-access-vz57q\") pod \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\" (UID: \"8bf370ba-e510-4fb6-8860-2372a1e53b2d\") "
Jul 31 12:48:09.302334 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.302102    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw48d\" (UniqueName: \"kubernetes.io/projected/abcaf387-d75b-4b76-b4f2-e2a721c129f6-kube-api-access-lw48d\") pod \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\" (UID: \"abcaf387-d75b-4b76-b4f2-e2a721c129f6\") "
Jul 31 12:48:09.302334 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.302249    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf370ba-e510-4fb6-8860-2372a1e53b2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bf370ba-e510-4fb6-8860-2372a1e53b2d" (UID: "8bf370ba-e510-4fb6-8860-2372a1e53b2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:48:09.302450 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.302386    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b87365-0cd5-4a26-9955-8ff168e70adb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46b87365-0cd5-4a26-9955-8ff168e70adb" (UID: "46b87365-0cd5-4a26-9955-8ff168e70adb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:48:09.302518 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.302496    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b87365-0cd5-4a26-9955-8ff168e70adb-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:09.302576 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.302525    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf370ba-e510-4fb6-8860-2372a1e53b2d-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:09.302652 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.302634    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcaf387-d75b-4b76-b4f2-e2a721c129f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abcaf387-d75b-4b76-b4f2-e2a721c129f6" (UID: "abcaf387-d75b-4b76-b4f2-e2a721c129f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:48:09.304158 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.304135    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf370ba-e510-4fb6-8860-2372a1e53b2d-kube-api-access-vz57q" (OuterVolumeSpecName: "kube-api-access-vz57q") pod "8bf370ba-e510-4fb6-8860-2372a1e53b2d" (UID: "8bf370ba-e510-4fb6-8860-2372a1e53b2d"). InnerVolumeSpecName "kube-api-access-vz57q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:48:09.304253 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.304189    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b87365-0cd5-4a26-9955-8ff168e70adb-kube-api-access-zpvcj" (OuterVolumeSpecName: "kube-api-access-zpvcj") pod "46b87365-0cd5-4a26-9955-8ff168e70adb" (UID: "46b87365-0cd5-4a26-9955-8ff168e70adb"). InnerVolumeSpecName "kube-api-access-zpvcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:48:09.304422 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.304396    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b87365-0cd5-4a26-9955-8ff168e70adb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46b87365-0cd5-4a26-9955-8ff168e70adb" (UID: "46b87365-0cd5-4a26-9955-8ff168e70adb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:48:09.304507 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.304420    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcaf387-d75b-4b76-b4f2-e2a721c129f6-kube-api-access-lw48d" (OuterVolumeSpecName: "kube-api-access-lw48d") pod "abcaf387-d75b-4b76-b4f2-e2a721c129f6" (UID: "abcaf387-d75b-4b76-b4f2-e2a721c129f6"). InnerVolumeSpecName "kube-api-access-lw48d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:48:09.403754 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.403734    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b87365-0cd5-4a26-9955-8ff168e70adb-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:09.403868 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.403756    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpvcj\" (UniqueName: \"kubernetes.io/projected/46b87365-0cd5-4a26-9955-8ff168e70adb-kube-api-access-zpvcj\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:09.403868 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.403767    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcaf387-d75b-4b76-b4f2-e2a721c129f6-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:09.403868 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.403779    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz57q\" (UniqueName: \"kubernetes.io/projected/8bf370ba-e510-4fb6-8860-2372a1e53b2d-kube-api-access-vz57q\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:09.403868 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.403787    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw48d\" (UniqueName: \"kubernetes.io/projected/abcaf387-d75b-4b76-b4f2-e2a721c129f6-kube-api-access-lw48d\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:09.947699 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.947662    2586 generic.go:356] "Generic (PLEG): container finished" podID="abcaf387-d75b-4b76-b4f2-e2a721c129f6" containerID="9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023" exitCode=0
Jul 31 12:48:09.948095 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.947706    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"
Jul 31 12:48:09.948095 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.947748    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw" event={"ID":"abcaf387-d75b-4b76-b4f2-e2a721c129f6","Type":"ContainerDied","Data":"9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023"}
Jul 31 12:48:09.948095 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.947801    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw" event={"ID":"abcaf387-d75b-4b76-b4f2-e2a721c129f6","Type":"ContainerDied","Data":"ffd03987892d669a3bd530c67d582d271997468c6cb1fe8dc00ca0ee23675907"}
Jul 31 12:48:09.948095 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.947828    2586 scope.go:117] "RemoveContainer" containerID="9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023"
Jul 31 12:48:09.949162 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.949142    2586 generic.go:356] "Generic (PLEG): container finished" podID="46b87365-0cd5-4a26-9955-8ff168e70adb" containerID="ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12" exitCode=0
Jul 31 12:48:09.949239 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.949209    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"
Jul 31 12:48:09.949239 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.949220    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2" event={"ID":"46b87365-0cd5-4a26-9955-8ff168e70adb","Type":"ContainerDied","Data":"ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12"}
Jul 31 12:48:09.949359 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.949286    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2" event={"ID":"46b87365-0cd5-4a26-9955-8ff168e70adb","Type":"ContainerDied","Data":"4ecd918774bd9caa81b69c766549ffbe35c11ed5e3058cd474c48d47d86c9fbc"}
Jul 31 12:48:09.950500 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.950473    2586 generic.go:356] "Generic (PLEG): container finished" podID="8bf370ba-e510-4fb6-8860-2372a1e53b2d" containerID="294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f" exitCode=0
Jul 31 12:48:09.950611 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.950512    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"
Jul 31 12:48:09.950611 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.950527    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j" event={"ID":"8bf370ba-e510-4fb6-8860-2372a1e53b2d","Type":"ContainerDied","Data":"294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f"}
Jul 31 12:48:09.950611 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.950568    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j" event={"ID":"8bf370ba-e510-4fb6-8860-2372a1e53b2d","Type":"ContainerDied","Data":"c89e024e2dcc81b3c572b8f8e3b7f48f0322070ff4605e44aecb561184add3d4"}
Jul 31 12:48:09.957953 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.957819    2586 scope.go:117] "RemoveContainer" containerID="9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023"
Jul 31 12:48:09.958241 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:09.958218    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023\": container with ID starting with 9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023 not found: ID does not exist" containerID="9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023"
Jul 31 12:48:09.958340 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.958251    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023"} err="failed to get container status \"9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023\": rpc error: code = NotFound desc = could not find container \"9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023\": container with ID starting with 9cfd6cae507b121be64819c2fe2d0017d3383029bcce0167a639ab7a4e8ab023 not found: ID does not exist"
Jul 31 12:48:09.958340 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.958301    2586 scope.go:117] "RemoveContainer" containerID="ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12"
Jul 31 12:48:09.966035 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.966016    2586 scope.go:117] "RemoveContainer" containerID="ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12"
Jul 31 12:48:09.966253 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:09.966236    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12\": container with ID starting with ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12 not found: ID does not exist" containerID="ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12"
Jul 31 12:48:09.966320 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.966258    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12"} err="failed to get container status \"ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12\": rpc error: code = NotFound desc = could not find container \"ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12\": container with ID starting with ab8837572ad472edd09525381c53e233145eb1f3dd45db6d2ddc5836d1f7ee12 not found: ID does not exist"
Jul 31 12:48:09.966320 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.966289    2586 scope.go:117] "RemoveContainer" containerID="294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f"
Jul 31 12:48:09.973945 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.973922    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"]
Jul 31 12:48:09.974300 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.974285    2586 scope.go:117] "RemoveContainer" containerID="294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f"
Jul 31 12:48:09.974568 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:09.974545    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f\": container with ID starting with 294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f not found: ID does not exist" containerID="294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f"
Jul 31 12:48:09.974709 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.974581    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f"} err="failed to get container status \"294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f\": rpc error: code = NotFound desc = could not find container \"294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f\": container with ID starting with 294a219cf5b259c0d0316245d312c166501976b21f071d57752575be17dd462f not found: ID does not exist"
Jul 31 12:48:09.976916 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.976897    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84c24c8b-68cc9674b8-xs5kw"]
Jul 31 12:48:09.986577 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.986555    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"]
Jul 31 12:48:09.995436 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:09.995418    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84c24c8b-8fcbccc49-j95s2"]
Jul 31 12:48:10.005093 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:10.005076    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"]
Jul 31 12:48:10.011176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:10.011153    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84c24c8b-6b7c67bf48-4nr8j"]
Jul 31 12:48:10.486113 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:10.486079    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b87365-0cd5-4a26-9955-8ff168e70adb" path="/var/lib/kubelet/pods/46b87365-0cd5-4a26-9955-8ff168e70adb/volumes"
Jul 31 12:48:10.486492 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:10.486478    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf370ba-e510-4fb6-8860-2372a1e53b2d" path="/var/lib/kubelet/pods/8bf370ba-e510-4fb6-8860-2372a1e53b2d/volumes"
Jul 31 12:48:10.486821 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:10.486808    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abcaf387-d75b-4b76-b4f2-e2a721c129f6" path="/var/lib/kubelet/pods/abcaf387-d75b-4b76-b4f2-e2a721c129f6/volumes"
Jul 31 12:48:12.982139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982090    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"]
Jul 31 12:48:12.982653 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982637    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b87365-0cd5-4a26-9955-8ff168e70adb" containerName="maas-api"
Jul 31 12:48:12.982732 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982656    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b87365-0cd5-4a26-9955-8ff168e70adb" containerName="maas-api"
Jul 31 12:48:12.982732 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982679    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abcaf387-d75b-4b76-b4f2-e2a721c129f6" containerName="payload-pre-processing"
Jul 31 12:48:12.982732 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982688    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="abcaf387-d75b-4b76-b4f2-e2a721c129f6" containerName="payload-pre-processing"
Jul 31 12:48:12.982732 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982722    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf370ba-e510-4fb6-8860-2372a1e53b2d" containerName="payload-processing"
Jul 31 12:48:12.982732 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982730    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf370ba-e510-4fb6-8860-2372a1e53b2d" containerName="payload-processing"
Jul 31 12:48:12.982973 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982812    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b87365-0cd5-4a26-9955-8ff168e70adb" containerName="maas-api"
Jul 31 12:48:12.982973 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982825    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf370ba-e510-4fb6-8860-2372a1e53b2d" containerName="payload-processing"
Jul 31 12:48:12.982973 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.982834    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="abcaf387-d75b-4b76-b4f2-e2a721c129f6" containerName="payload-pre-processing"
Jul 31 12:48:12.987362 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.987340    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:12.989944 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.989914    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-643d0a88-serving-cert\""
Jul 31 12:48:12.996351 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:12.995958    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"]
Jul 31 12:48:13.139090 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.139058    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c933-c418-4edb-939b-2c33ab94511c-openshift-service-ca\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.139259 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.139200    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-856rl\" (UniqueName: \"kubernetes.io/projected/5923c933-c418-4edb-939b-2c33ab94511c-kube-api-access-856rl\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.139259 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.139229    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c933-c418-4edb-939b-2c33ab94511c-maas-api-tls\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.240500 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.240412    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-856rl\" (UniqueName: \"kubernetes.io/projected/5923c933-c418-4edb-939b-2c33ab94511c-kube-api-access-856rl\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.240500 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.240460    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c933-c418-4edb-939b-2c33ab94511c-maas-api-tls\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.240691 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.240508    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c933-c418-4edb-939b-2c33ab94511c-openshift-service-ca\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.240691 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:13.240636    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-serving-cert: secret "maas-api-e2e-migrate-643d0a88-serving-cert" not found
Jul 31 12:48:13.240757 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:13.240709    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5923c933-c418-4edb-939b-2c33ab94511c-maas-api-tls podName:5923c933-c418-4edb-939b-2c33ab94511c nodeName:}" failed. No retries permitted until 2026-07-31 12:48:13.740693481 +0000 UTC m=+1981.727914999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5923c933-c418-4edb-939b-2c33ab94511c-maas-api-tls") pod "maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm" (UID: "5923c933-c418-4edb-939b-2c33ab94511c") : secret "maas-api-e2e-migrate-643d0a88-serving-cert" not found
Jul 31 12:48:13.241276 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.241244    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c933-c418-4edb-939b-2c33ab94511c-openshift-service-ca\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.248846 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.248819    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-856rl\" (UniqueName: \"kubernetes.io/projected/5923c933-c418-4edb-939b-2c33ab94511c-kube-api-access-856rl\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.737770 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.737734    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"]
Jul 31 12:48:13.741427 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.741405    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:13.743964 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.743925    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c933-c418-4edb-939b-2c33ab94511c-maas-api-tls\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.744063 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.743971    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-643d0a88\""
Jul 31 12:48:13.744063 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.744045    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-643d0a88-dockercfg-jmvd8\""
Jul 31 12:48:13.746511 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.746492    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c933-c418-4edb-939b-2c33ab94511c-maas-api-tls\") pod \"maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.752472 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.752453    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"]
Jul 31 12:48:13.759812 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.759793    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"]
Jul 31 12:48:13.763827 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.763812    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:13.771828 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.771807    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"]
Jul 31 12:48:13.845341 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.845304    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5p4\" (UniqueName: \"kubernetes.io/projected/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-kube-api-access-vb5p4\") pod \"payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8\" (UID: \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\") " pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:13.845508 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.845358    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvxj6\" (UniqueName: \"kubernetes.io/projected/7372afa4-5cad-415e-a799-9ab0ab7c38a1-kube-api-access-tvxj6\") pod \"payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs\" (UID: \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:13.845558 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.845501    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372afa4-5cad-415e-a799-9ab0ab7c38a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs\" (UID: \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:13.845601 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.845534    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-plugins-config-volume\") pod \"payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8\" (UID: \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\") " pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:13.900742 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.900713    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:13.947094 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.947058    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5p4\" (UniqueName: \"kubernetes.io/projected/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-kube-api-access-vb5p4\") pod \"payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8\" (UID: \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\") " pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:13.947306 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.947102    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvxj6\" (UniqueName: \"kubernetes.io/projected/7372afa4-5cad-415e-a799-9ab0ab7c38a1-kube-api-access-tvxj6\") pod \"payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs\" (UID: \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:13.947306 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.947176    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372afa4-5cad-415e-a799-9ab0ab7c38a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs\" (UID: \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:13.947306 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.947206    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-plugins-config-volume\") pod \"payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8\" (UID: \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\") " pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:13.948078 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.948047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-plugins-config-volume\") pod \"payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8\" (UID: \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\") " pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:13.948501 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.948477    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372afa4-5cad-415e-a799-9ab0ab7c38a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs\" (UID: \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:13.960212 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.958101    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5p4\" (UniqueName: \"kubernetes.io/projected/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-kube-api-access-vb5p4\") pod \"payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8\" (UID: \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\") " pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:13.960848 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:13.960657    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvxj6\" (UniqueName: \"kubernetes.io/projected/7372afa4-5cad-415e-a799-9ab0ab7c38a1-kube-api-access-tvxj6\") pod \"payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs\" (UID: \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:14.038351 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.038322    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"]
Jul 31 12:48:14.040785 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:48:14.040752    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5923c933_c418_4edb_939b_2c33ab94511c.slice/crio-59b86c8f033ac23d1e3f32d8e38018dbc38569f25cf0ff7e3d080abcccbf4123 WatchSource:0}: Error finding container 59b86c8f033ac23d1e3f32d8e38018dbc38569f25cf0ff7e3d080abcccbf4123: Status 404 returned error can't find the container with id 59b86c8f033ac23d1e3f32d8e38018dbc38569f25cf0ff7e3d080abcccbf4123
Jul 31 12:48:14.062545 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.062522    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:14.077872 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.077847    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:14.224912 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.224886    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"]
Jul 31 12:48:14.226705 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:48:14.226682    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9144a2b9_e05a_4718_9cb1_8b8ab2ba08db.slice/crio-17ac7e103f88ac1e15eee4ac3a74c2c8651df46054f0546508a3e43f187bda18 WatchSource:0}: Error finding container 17ac7e103f88ac1e15eee4ac3a74c2c8651df46054f0546508a3e43f187bda18: Status 404 returned error can't find the container with id 17ac7e103f88ac1e15eee4ac3a74c2c8651df46054f0546508a3e43f187bda18
Jul 31 12:48:14.406059 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.406030    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"]
Jul 31 12:48:14.408176 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:48:14.408149    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7372afa4_5cad_415e_a799_9ab0ab7c38a1.slice/crio-e057037eb4bf7992d7059dd24aa7601ee53a2711a8062f11fc741d93109295c5 WatchSource:0}: Error finding container e057037eb4bf7992d7059dd24aa7601ee53a2711a8062f11fc741d93109295c5: Status 404 returned error can't find the container with id e057037eb4bf7992d7059dd24aa7601ee53a2711a8062f11fc741d93109295c5
Jul 31 12:48:14.522247 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.522222    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:48:14.653758 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.653720    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-credential-socket\") pod \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") "
Jul 31 12:48:14.653930 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.653779    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istiod-ca-cert\") pod \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") "
Jul 31 12:48:14.653930 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.653819    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-certs\") pod \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") "
Jul 31 12:48:14.653930 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.653889    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-data\") pod \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") "
Jul 31 12:48:14.653930 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.653916    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-token\") pod \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") "
Jul 31 12:48:14.654163 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.653977    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcrbr\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-kube-api-access-xcrbr\") pod \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") "
Jul 31 12:48:14.654163 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.653995    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd37787f-4bd7-4f21-b4d0-3337cb79faa8" (UID: "cd37787f-4bd7-4f21-b4d0-3337cb79faa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:48:14.654163 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.654003    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-envoy\") pod \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") "
Jul 31 12:48:14.654163 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.654030    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-socket\") pod \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") "
Jul 31 12:48:14.654163 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.654065    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-podinfo\") pod \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\" (UID: \"cd37787f-4bd7-4f21-b4d0-3337cb79faa8\") "
Jul 31 12:48:14.654163 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.654141    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd37787f-4bd7-4f21-b4d0-3337cb79faa8" (UID: "cd37787f-4bd7-4f21-b4d0-3337cb79faa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:48:14.654480 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.654319    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd37787f-4bd7-4f21-b4d0-3337cb79faa8" (UID: "cd37787f-4bd7-4f21-b4d0-3337cb79faa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:48:14.654480 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.654416    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:14.654480 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.654433    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:14.654480 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.654446    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:14.655155 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.655095    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd37787f-4bd7-4f21-b4d0-3337cb79faa8" (UID: "cd37787f-4bd7-4f21-b4d0-3337cb79faa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:48:14.655332 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.655308    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd37787f-4bd7-4f21-b4d0-3337cb79faa8" (UID: "cd37787f-4bd7-4f21-b4d0-3337cb79faa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:48:14.656466 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.656435    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd37787f-4bd7-4f21-b4d0-3337cb79faa8" (UID: "cd37787f-4bd7-4f21-b4d0-3337cb79faa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 12:48:14.656693 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.656668    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-kube-api-access-xcrbr" (OuterVolumeSpecName: "kube-api-access-xcrbr") pod "cd37787f-4bd7-4f21-b4d0-3337cb79faa8" (UID: "cd37787f-4bd7-4f21-b4d0-3337cb79faa8"). InnerVolumeSpecName "kube-api-access-xcrbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:48:14.657182 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.657164    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd37787f-4bd7-4f21-b4d0-3337cb79faa8" (UID: "cd37787f-4bd7-4f21-b4d0-3337cb79faa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:48:14.657182 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.657165    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd37787f-4bd7-4f21-b4d0-3337cb79faa8" (UID: "cd37787f-4bd7-4f21-b4d0-3337cb79faa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:48:14.755923 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.755889    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcrbr\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-kube-api-access-xcrbr\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:14.755923 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.755916    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:14.755923 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.755926    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:14.756143 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.755934    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:14.756143 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.755942    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:14.756143 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.755950    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd37787f-4bd7-4f21-b4d0-3337cb79faa8-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:14.979185 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.979147    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs" event={"ID":"7372afa4-5cad-415e-a799-9ab0ab7c38a1","Type":"ContainerStarted","Data":"ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff"}
Jul 31 12:48:14.979396 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.979201    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs" event={"ID":"7372afa4-5cad-415e-a799-9ab0ab7c38a1","Type":"ContainerStarted","Data":"e057037eb4bf7992d7059dd24aa7601ee53a2711a8062f11fc741d93109295c5"}
Jul 31 12:48:14.979396 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.979260    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:14.980874 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.980844    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8" event={"ID":"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db","Type":"ContainerStarted","Data":"d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959"}
Jul 31 12:48:14.980999 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.980893    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8" event={"ID":"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db","Type":"ContainerStarted","Data":"17ac7e103f88ac1e15eee4ac3a74c2c8651df46054f0546508a3e43f187bda18"}
Jul 31 12:48:14.980999 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.980938    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:14.982557 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.982533    2586 generic.go:356] "Generic (PLEG): container finished" podID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerID="5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f" exitCode=0
Jul 31 12:48:14.982683 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.982587    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"
Jul 31 12:48:14.982683 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.982612    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" event={"ID":"cd37787f-4bd7-4f21-b4d0-3337cb79faa8","Type":"ContainerDied","Data":"5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f"}
Jul 31 12:48:14.982683 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.982650    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx" event={"ID":"cd37787f-4bd7-4f21-b4d0-3337cb79faa8","Type":"ContainerDied","Data":"0a7079fd9fcff2652c4670cdf93e1737e09dc5002015098c0ed64ed5e0a05e14"}
Jul 31 12:48:14.982683 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.982676    2586 scope.go:117] "RemoveContainer" containerID="5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f"
Jul 31 12:48:14.984323 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.984300    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm" event={"ID":"5923c933-c418-4edb-939b-2c33ab94511c","Type":"ContainerStarted","Data":"1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c"}
Jul 31 12:48:14.984445 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.984347    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm" event={"ID":"5923c933-c418-4edb-939b-2c33ab94511c","Type":"ContainerStarted","Data":"59b86c8f033ac23d1e3f32d8e38018dbc38569f25cf0ff7e3d080abcccbf4123"}
Jul 31 12:48:14.984445 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.984379    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:14.991523 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.991499    2586 scope.go:117] "RemoveContainer" containerID="8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476"
Jul 31 12:48:14.996862 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.996821    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs" podStartSLOduration=1.996808847 podStartE2EDuration="1.996808847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:48:14.995303413 +0000 UTC m=+1982.982524956" watchObservedRunningTime="2026-07-31 12:48:14.996808847 +0000 UTC m=+1982.984030386"
Jul 31 12:48:14.999649 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.999631    2586 scope.go:117] "RemoveContainer" containerID="5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f"
Jul 31 12:48:14.999876 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:14.999858    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f\": container with ID starting with 5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f not found: ID does not exist" containerID="5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f"
Jul 31 12:48:14.999947 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.999888    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f"} err="failed to get container status \"5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f\": rpc error: code = NotFound desc = could not find container \"5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f\": container with ID starting with 5e4b22bd9ecf05b79466ae5e37b4203c4dd4a424985daeb9be26c0948cec3a4f not found: ID does not exist"
Jul 31 12:48:14.999947 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:14.999914    2586 scope.go:117] "RemoveContainer" containerID="8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476"
Jul 31 12:48:15.000147 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:15.000130    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476\": container with ID starting with 8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476 not found: ID does not exist" containerID="8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476"
Jul 31 12:48:15.000182 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:15.000153    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476"} err="failed to get container status \"8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476\": rpc error: code = NotFound desc = could not find container \"8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476\": container with ID starting with 8a20717ba3fce3f6953e197bb70bf476b59ee9f24ef1761ed18805118ac45476 not found: ID does not exist"
Jul 31 12:48:15.017334 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:15.014033    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8" podStartSLOduration=2.014016671 podStartE2EDuration="2.014016671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:48:15.011383017 +0000 UTC m=+1982.998604559" watchObservedRunningTime="2026-07-31 12:48:15.014016671 +0000 UTC m=+1983.001238216"
Jul 31 12:48:15.028358 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:15.028330    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"]
Jul 31 12:48:15.034013 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:15.033991    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-84c24c8b-openshift-default-58c77b4f4d-fkvsx"]
Jul 31 12:48:15.050869 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:15.050826    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm" podStartSLOduration=2.604267192 podStartE2EDuration="3.050813003s" totalImagesPullingTime="446.545811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:48:15.048776414 +0000 UTC m=+1983.035997978" watchObservedRunningTime="2026-07-31 12:48:15.050813003 +0000 UTC m=+1983.038034585"
Jul 31 12:48:16.485213 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:16.485182    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" path="/var/lib/kubelet/pods/cd37787f-4bd7-4f21-b4d0-3337cb79faa8/volumes"
Jul 31 12:48:20.995391 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:20.995362    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:25.991912 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:25.991886    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:25.992347 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:25.991951    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:44.248728 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.248694    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"]
Jul 31 12:48:44.249221 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.248929    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm" podUID="5923c933-c418-4edb-939b-2c33ab94511c" containerName="maas-api" containerID="cri-o://1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c" gracePeriod=30
Jul 31 12:48:44.306889 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.306855    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"]
Jul 31 12:48:44.307118 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.307094    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8" podUID="9144a2b9-e05a-4718-9cb1-8b8ab2ba08db" containerName="payload-processing" containerID="cri-o://d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959" gracePeriod=30
Jul 31 12:48:44.314055 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.314022    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"]
Jul 31 12:48:44.314313 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.314284    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs" podUID="7372afa4-5cad-415e-a799-9ab0ab7c38a1" containerName="payload-pre-processing" containerID="cri-o://ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff" gracePeriod=30
Jul 31 12:48:44.607111 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.607084    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:44.611757 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.611736    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:44.619466 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.619447    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:44.702675 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.702647    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-plugins-config-volume\") pod \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\" (UID: \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\") "
Jul 31 12:48:44.702836 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.702680    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-856rl\" (UniqueName: \"kubernetes.io/projected/5923c933-c418-4edb-939b-2c33ab94511c-kube-api-access-856rl\") pod \"5923c933-c418-4edb-939b-2c33ab94511c\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") "
Jul 31 12:48:44.702836 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.702720    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c933-c418-4edb-939b-2c33ab94511c-openshift-service-ca\") pod \"5923c933-c418-4edb-939b-2c33ab94511c\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") "
Jul 31 12:48:44.702836 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.702746    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb5p4\" (UniqueName: \"kubernetes.io/projected/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-kube-api-access-vb5p4\") pod \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\" (UID: \"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db\") "
Jul 31 12:48:44.702836 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.702822    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c933-c418-4edb-939b-2c33ab94511c-maas-api-tls\") pod \"5923c933-c418-4edb-939b-2c33ab94511c\" (UID: \"5923c933-c418-4edb-939b-2c33ab94511c\") "
Jul 31 12:48:44.703050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.702862    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvxj6\" (UniqueName: \"kubernetes.io/projected/7372afa4-5cad-415e-a799-9ab0ab7c38a1-kube-api-access-tvxj6\") pod \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\" (UID: \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\") "
Jul 31 12:48:44.703050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.702895    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372afa4-5cad-415e-a799-9ab0ab7c38a1-plugins-config-volume\") pod \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\" (UID: \"7372afa4-5cad-415e-a799-9ab0ab7c38a1\") "
Jul 31 12:48:44.703166 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.703089    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5923c933-c418-4edb-939b-2c33ab94511c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5923c933-c418-4edb-939b-2c33ab94511c" (UID: "5923c933-c418-4edb-939b-2c33ab94511c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:48:44.703166 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.703090    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9144a2b9-e05a-4718-9cb1-8b8ab2ba08db" (UID: "9144a2b9-e05a-4718-9cb1-8b8ab2ba08db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:48:44.703389 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.703365    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:44.703510 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.703394    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c933-c418-4edb-939b-2c33ab94511c-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:44.703510 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.703452    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7372afa4-5cad-415e-a799-9ab0ab7c38a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7372afa4-5cad-415e-a799-9ab0ab7c38a1" (UID: "7372afa4-5cad-415e-a799-9ab0ab7c38a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:48:44.705391 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.705365    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7372afa4-5cad-415e-a799-9ab0ab7c38a1-kube-api-access-tvxj6" (OuterVolumeSpecName: "kube-api-access-tvxj6") pod "7372afa4-5cad-415e-a799-9ab0ab7c38a1" (UID: "7372afa4-5cad-415e-a799-9ab0ab7c38a1"). InnerVolumeSpecName "kube-api-access-tvxj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:48:44.705391 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.705377    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5923c933-c418-4edb-939b-2c33ab94511c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5923c933-c418-4edb-939b-2c33ab94511c" (UID: "5923c933-c418-4edb-939b-2c33ab94511c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:48:44.705391 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.705376    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-kube-api-access-vb5p4" (OuterVolumeSpecName: "kube-api-access-vb5p4") pod "9144a2b9-e05a-4718-9cb1-8b8ab2ba08db" (UID: "9144a2b9-e05a-4718-9cb1-8b8ab2ba08db"). InnerVolumeSpecName "kube-api-access-vb5p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:48:44.705545 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.705410    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5923c933-c418-4edb-939b-2c33ab94511c-kube-api-access-856rl" (OuterVolumeSpecName: "kube-api-access-856rl") pod "5923c933-c418-4edb-939b-2c33ab94511c" (UID: "5923c933-c418-4edb-939b-2c33ab94511c"). InnerVolumeSpecName "kube-api-access-856rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:48:44.804494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.804408    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5p4\" (UniqueName: \"kubernetes.io/projected/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db-kube-api-access-vb5p4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:44.804494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.804435    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c933-c418-4edb-939b-2c33ab94511c-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:44.804494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.804450    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvxj6\" (UniqueName: \"kubernetes.io/projected/7372afa4-5cad-415e-a799-9ab0ab7c38a1-kube-api-access-tvxj6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:44.804494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.804467    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372afa4-5cad-415e-a799-9ab0ab7c38a1-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:44.804494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:44.804479    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-856rl\" (UniqueName: \"kubernetes.io/projected/5923c933-c418-4edb-939b-2c33ab94511c-kube-api-access-856rl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:48:45.127701 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.127611    2586 generic.go:356] "Generic (PLEG): container finished" podID="5923c933-c418-4edb-939b-2c33ab94511c" containerID="1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c" exitCode=0
Jul 31 12:48:45.127701 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.127656    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm" event={"ID":"5923c933-c418-4edb-939b-2c33ab94511c","Type":"ContainerDied","Data":"1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c"}
Jul 31 12:48:45.127701 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.127677    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"
Jul 31 12:48:45.127950 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.127753    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm" event={"ID":"5923c933-c418-4edb-939b-2c33ab94511c","Type":"ContainerDied","Data":"59b86c8f033ac23d1e3f32d8e38018dbc38569f25cf0ff7e3d080abcccbf4123"}
Jul 31 12:48:45.127950 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.127784    2586 scope.go:117] "RemoveContainer" containerID="1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c"
Jul 31 12:48:45.129114 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.129093    2586 generic.go:356] "Generic (PLEG): container finished" podID="7372afa4-5cad-415e-a799-9ab0ab7c38a1" containerID="ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff" exitCode=0
Jul 31 12:48:45.129228 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.129162    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"
Jul 31 12:48:45.129228 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.129168    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs" event={"ID":"7372afa4-5cad-415e-a799-9ab0ab7c38a1","Type":"ContainerDied","Data":"ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff"}
Jul 31 12:48:45.129228 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.129213    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs" event={"ID":"7372afa4-5cad-415e-a799-9ab0ab7c38a1","Type":"ContainerDied","Data":"e057037eb4bf7992d7059dd24aa7601ee53a2711a8062f11fc741d93109295c5"}
Jul 31 12:48:45.131498 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.131475    2586 generic.go:356] "Generic (PLEG): container finished" podID="9144a2b9-e05a-4718-9cb1-8b8ab2ba08db" containerID="d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959" exitCode=0
Jul 31 12:48:45.131631 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.131512    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8" event={"ID":"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db","Type":"ContainerDied","Data":"d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959"}
Jul 31 12:48:45.131631 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.131540    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"
Jul 31 12:48:45.131631 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.131577    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8" event={"ID":"9144a2b9-e05a-4718-9cb1-8b8ab2ba08db","Type":"ContainerDied","Data":"17ac7e103f88ac1e15eee4ac3a74c2c8651df46054f0546508a3e43f187bda18"}
Jul 31 12:48:45.139339 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.139324    2586 scope.go:117] "RemoveContainer" containerID="1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c"
Jul 31 12:48:45.139570 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:45.139554    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c\": container with ID starting with 1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c not found: ID does not exist" containerID="1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c"
Jul 31 12:48:45.139621 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.139578    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c"} err="failed to get container status \"1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c\": rpc error: code = NotFound desc = could not find container \"1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c\": container with ID starting with 1dcc81607c4a6a6a554165075224ee6cffc7b79cbeabdf93448f78be1f8e6f5c not found: ID does not exist"
Jul 31 12:48:45.139621 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.139601    2586 scope.go:117] "RemoveContainer" containerID="ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff"
Jul 31 12:48:45.147439 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.147424    2586 scope.go:117] "RemoveContainer" containerID="ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff"
Jul 31 12:48:45.147690 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:45.147672    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff\": container with ID starting with ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff not found: ID does not exist" containerID="ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff"
Jul 31 12:48:45.147749 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.147698    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff"} err="failed to get container status \"ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff\": rpc error: code = NotFound desc = could not find container \"ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff\": container with ID starting with ae8b6e921000a65468557853732ba5575ebac8b4d2d756e7a1bca74a1d18daff not found: ID does not exist"
Jul 31 12:48:45.147749 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.147714    2586 scope.go:117] "RemoveContainer" containerID="d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959"
Jul 31 12:48:45.151998 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.151979    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"]
Jul 31 12:48:45.155450 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.155426    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-643d0a88-569bf79d56-qb2lm"]
Jul 31 12:48:45.156121 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.156102    2586 scope.go:117] "RemoveContainer" containerID="d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959"
Jul 31 12:48:45.156359 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:48:45.156341    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959\": container with ID starting with d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959 not found: ID does not exist" containerID="d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959"
Jul 31 12:48:45.156432 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.156368    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959"} err="failed to get container status \"d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959\": rpc error: code = NotFound desc = could not find container \"d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959\": container with ID starting with d14ca6d3f2594d00a03117871db6650f2b7977608c0c68e0ec2145c473ca7959 not found: ID does not exist"
Jul 31 12:48:45.165121 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.165101    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"]
Jul 31 12:48:45.168462 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.168445    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-643d0a88-5c54467d4d-74fvs"]
Jul 31 12:48:45.178951 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.178933    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"]
Jul 31 12:48:45.182294 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.182275    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-643d0a88-86ddbd7ff7-jx8j8"]
Jul 31 12:48:45.468377 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.468339    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"]
Jul 31 12:48:45.468883 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.468865    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7372afa4-5cad-415e-a799-9ab0ab7c38a1" containerName="payload-pre-processing"
Jul 31 12:48:45.468955 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.468886    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7372afa4-5cad-415e-a799-9ab0ab7c38a1" containerName="payload-pre-processing"
Jul 31 12:48:45.468955 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.468898    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5923c933-c418-4edb-939b-2c33ab94511c" containerName="maas-api"
Jul 31 12:48:45.468955 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.468905    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5923c933-c418-4edb-939b-2c33ab94511c" containerName="maas-api"
Jul 31 12:48:45.468955 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.468917    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy"
Jul 31 12:48:45.468955 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.468926    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy"
Jul 31 12:48:45.468955 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.468944    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9144a2b9-e05a-4718-9cb1-8b8ab2ba08db" containerName="payload-processing"
Jul 31 12:48:45.468955 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.468955    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9144a2b9-e05a-4718-9cb1-8b8ab2ba08db" containerName="payload-processing"
Jul 31 12:48:45.469307 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.469047    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy"
Jul 31 12:48:45.469307 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.469058    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7372afa4-5cad-415e-a799-9ab0ab7c38a1" containerName="payload-pre-processing"
Jul 31 12:48:45.469307 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.469069    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy"
Jul 31 12:48:45.469307 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.469079    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9144a2b9-e05a-4718-9cb1-8b8ab2ba08db" containerName="payload-processing"
Jul 31 12:48:45.469307 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.469091    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5923c933-c418-4edb-939b-2c33ab94511c" containerName="maas-api"
Jul 31 12:48:45.473801 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.473778    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.476879 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.476858    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8fed071b-openshift-default-dockercfg-k7g8z\""
Jul 31 12:48:45.484462 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.484440    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"]
Jul 31 12:48:45.510921 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.510870    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-socket\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.510921 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.510903    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-istio-token\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.511138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.511004    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-envoy\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.511138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.511053    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l89h\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-kube-api-access-8l89h\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.511138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.511081    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-certs\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.511138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.511125    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae6299a-e317-4294-a257-b4ae9b353c86-istio-podinfo\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.511388 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.511145    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae6299a-e317-4294-a257-b4ae9b353c86-istiod-ca-cert\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.511388 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.511191    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-data\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.511388 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.511238    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-credential-socket\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.612541 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612504    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-socket\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.612541 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612546    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-istio-token\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.612817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612603    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-envoy\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.612817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612628    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l89h\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-kube-api-access-8l89h\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.612817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612657    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-certs\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.612817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612705    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae6299a-e317-4294-a257-b4ae9b353c86-istio-podinfo\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.612817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612726    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae6299a-e317-4294-a257-b4ae9b353c86-istiod-ca-cert\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.612817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612762    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-data\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.612817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612809    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-credential-socket\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.613101 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.612966    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-socket\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.613436 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.613409    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-certs\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.613559 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.613451    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-data\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.613559 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.613490    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-credential-socket\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.613559 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.613523    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae6299a-e317-4294-a257-b4ae9b353c86-istiod-ca-cert\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.615615 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.615594    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-envoy\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.615772 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.615758    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae6299a-e317-4294-a257-b4ae9b353c86-istio-podinfo\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.624827 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.624802    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l89h\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-kube-api-access-8l89h\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.626785 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.626768    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-istio-token\") pod \"e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") " pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.786867 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.786781    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:45.919684 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.919658    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"]
Jul 31 12:48:45.922345 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:48:45.922319    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae6299a_e317_4294_a257_b4ae9b353c86.slice/crio-ebbb90bab5cbaac4159cd847bfb6f58fa8f2501f3b42f95be8bcf6f0a5bb1ee4 WatchSource:0}: Error finding container ebbb90bab5cbaac4159cd847bfb6f58fa8f2501f3b42f95be8bcf6f0a5bb1ee4: Status 404 returned error can't find the container with id ebbb90bab5cbaac4159cd847bfb6f58fa8f2501f3b42f95be8bcf6f0a5bb1ee4
Jul 31 12:48:45.924474 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.924441    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:48:45.924583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.924519    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:48:45.924583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:45.924571    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:48:46.139817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:46.139718    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" event={"ID":"eae6299a-e317-4294-a257-b4ae9b353c86","Type":"ContainerStarted","Data":"1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66"}
Jul 31 12:48:46.139817 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:46.139789    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" event={"ID":"eae6299a-e317-4294-a257-b4ae9b353c86","Type":"ContainerStarted","Data":"ebbb90bab5cbaac4159cd847bfb6f58fa8f2501f3b42f95be8bcf6f0a5bb1ee4"}
Jul 31 12:48:46.161214 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:46.161167    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" podStartSLOduration=1.161150011 podStartE2EDuration="1.161150011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:48:46.158589961 +0000 UTC m=+2014.145811502" watchObservedRunningTime="2026-07-31 12:48:46.161150011 +0000 UTC m=+2014.148371550"
Jul 31 12:48:46.485234 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:46.485197    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5923c933-c418-4edb-939b-2c33ab94511c" path="/var/lib/kubelet/pods/5923c933-c418-4edb-939b-2c33ab94511c/volumes"
Jul 31 12:48:46.485700 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:46.485685    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7372afa4-5cad-415e-a799-9ab0ab7c38a1" path="/var/lib/kubelet/pods/7372afa4-5cad-415e-a799-9ab0ab7c38a1/volumes"
Jul 31 12:48:46.486004 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:46.485993    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9144a2b9-e05a-4718-9cb1-8b8ab2ba08db" path="/var/lib/kubelet/pods/9144a2b9-e05a-4718-9cb1-8b8ab2ba08db/volumes"
Jul 31 12:48:46.787396 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:46.787318    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:46.792116 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:46.792094    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:47.146805 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:47.146721    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:47.147686 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:47.147663    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:48:48.707357 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.707327    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"]
Jul 31 12:48:48.707726 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.707713    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy"
Jul 31 12:48:48.707771 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.707728    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd37787f-4bd7-4f21-b4d0-3337cb79faa8" containerName="istio-proxy"
Jul 31 12:48:48.712091 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.712067    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:48.714625 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.714605    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8fed071b-serving-cert\""
Jul 31 12:48:48.721663 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.721467    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"]
Jul 31 12:48:48.739472 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.739446    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de11ce2a-0a8f-482c-9912-2c246d9591df-openshift-service-ca\") pod \"maas-api-e2e-ait-8fed071b-85d474fd74-46zl5\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:48.739608 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.739482    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpvq\" (UniqueName: \"kubernetes.io/projected/de11ce2a-0a8f-482c-9912-2c246d9591df-kube-api-access-vjpvq\") pod \"maas-api-e2e-ait-8fed071b-85d474fd74-46zl5\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:48.739675 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.739605    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de11ce2a-0a8f-482c-9912-2c246d9591df-maas-api-tls\") pod \"maas-api-e2e-ait-8fed071b-85d474fd74-46zl5\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:48.840364 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.840332    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de11ce2a-0a8f-482c-9912-2c246d9591df-maas-api-tls\") pod \"maas-api-e2e-ait-8fed071b-85d474fd74-46zl5\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:48.840537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.840375    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de11ce2a-0a8f-482c-9912-2c246d9591df-openshift-service-ca\") pod \"maas-api-e2e-ait-8fed071b-85d474fd74-46zl5\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:48.840537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.840398    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpvq\" (UniqueName: \"kubernetes.io/projected/de11ce2a-0a8f-482c-9912-2c246d9591df-kube-api-access-vjpvq\") pod \"maas-api-e2e-ait-8fed071b-85d474fd74-46zl5\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:48.840962 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.840942    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de11ce2a-0a8f-482c-9912-2c246d9591df-openshift-service-ca\") pod \"maas-api-e2e-ait-8fed071b-85d474fd74-46zl5\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:48.842815 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.842790    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de11ce2a-0a8f-482c-9912-2c246d9591df-maas-api-tls\") pod \"maas-api-e2e-ait-8fed071b-85d474fd74-46zl5\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:48.849679 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:48.849653    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpvq\" (UniqueName: \"kubernetes.io/projected/de11ce2a-0a8f-482c-9912-2c246d9591df-kube-api-access-vjpvq\") pod \"maas-api-e2e-ait-8fed071b-85d474fd74-46zl5\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:49.026295 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.026188    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:49.163709 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.163686    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"]
Jul 31 12:48:49.165176 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:48:49.165150    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde11ce2a_0a8f_482c_9912_2c246d9591df.slice/crio-039e27bc851fdfa12f2325b5db497802a76e864da81b13efe2562fe336a79998 WatchSource:0}: Error finding container 039e27bc851fdfa12f2325b5db497802a76e864da81b13efe2562fe336a79998: Status 404 returned error can't find the container with id 039e27bc851fdfa12f2325b5db497802a76e864da81b13efe2562fe336a79998
Jul 31 12:48:49.351530 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.351445    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"]
Jul 31 12:48:49.356012 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.355989    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:48:49.360854 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.360834    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8fed071b\""
Jul 31 12:48:49.361086 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.361069    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8fed071b-dockercfg-xtgz8\""
Jul 31 12:48:49.375891 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.375869    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"]
Jul 31 12:48:49.402227 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.402205    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"]
Jul 31 12:48:49.405527 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.405506    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:48:49.415454 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.415423    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"]
Jul 31 12:48:49.445584 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.445557    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2zrq\" (UniqueName: \"kubernetes.io/projected/441df164-4272-47c7-a81e-eec2e94c6344-kube-api-access-j2zrq\") pod \"payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4\" (UID: \"441df164-4272-47c7-a81e-eec2e94c6344\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:48:49.445715 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.445587    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4b174-0878-4db6-b106-02a6166a9fb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w\" (UID: \"b4e4b174-0878-4db6-b106-02a6166a9fb8\") " pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:48:49.445715 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.445653    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441df164-4272-47c7-a81e-eec2e94c6344-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4\" (UID: \"441df164-4272-47c7-a81e-eec2e94c6344\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:48:49.445715 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.445693    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7zb\" (UniqueName: \"kubernetes.io/projected/b4e4b174-0878-4db6-b106-02a6166a9fb8-kube-api-access-qp7zb\") pod \"payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w\" (UID: \"b4e4b174-0878-4db6-b106-02a6166a9fb8\") " pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:48:49.546793 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.546757    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2zrq\" (UniqueName: \"kubernetes.io/projected/441df164-4272-47c7-a81e-eec2e94c6344-kube-api-access-j2zrq\") pod \"payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4\" (UID: \"441df164-4272-47c7-a81e-eec2e94c6344\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:48:49.546793 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.546792    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4b174-0878-4db6-b106-02a6166a9fb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w\" (UID: \"b4e4b174-0878-4db6-b106-02a6166a9fb8\") " pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:48:49.547029 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.546826    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441df164-4272-47c7-a81e-eec2e94c6344-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4\" (UID: \"441df164-4272-47c7-a81e-eec2e94c6344\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:48:49.547029 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.546869    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7zb\" (UniqueName: \"kubernetes.io/projected/b4e4b174-0878-4db6-b106-02a6166a9fb8-kube-api-access-qp7zb\") pod \"payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w\" (UID: \"b4e4b174-0878-4db6-b106-02a6166a9fb8\") " pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:48:49.547596 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.547571    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4b174-0878-4db6-b106-02a6166a9fb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w\" (UID: \"b4e4b174-0878-4db6-b106-02a6166a9fb8\") " pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:48:49.547692 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.547579    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441df164-4272-47c7-a81e-eec2e94c6344-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4\" (UID: \"441df164-4272-47c7-a81e-eec2e94c6344\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:48:49.555097 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.555077    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2zrq\" (UniqueName: \"kubernetes.io/projected/441df164-4272-47c7-a81e-eec2e94c6344-kube-api-access-j2zrq\") pod \"payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4\" (UID: \"441df164-4272-47c7-a81e-eec2e94c6344\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:48:49.555253 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.555237    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7zb\" (UniqueName: \"kubernetes.io/projected/b4e4b174-0878-4db6-b106-02a6166a9fb8-kube-api-access-qp7zb\") pod \"payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w\" (UID: \"b4e4b174-0878-4db6-b106-02a6166a9fb8\") " pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:48:49.666906 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.666878    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:48:49.716147 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.716116    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:48:49.875022 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:49.875001    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"]
Jul 31 12:48:49.876838 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:48:49.876799    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441df164_4272_47c7_a81e_eec2e94c6344.slice/crio-11e7ebe48be8a227c2b61a4016a0fd8fecefeb662784ec296ea07830031519e1 WatchSource:0}: Error finding container 11e7ebe48be8a227c2b61a4016a0fd8fecefeb662784ec296ea07830031519e1: Status 404 returned error can't find the container with id 11e7ebe48be8a227c2b61a4016a0fd8fecefeb662784ec296ea07830031519e1
Jul 31 12:48:50.016060 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.016032    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"]
Jul 31 12:48:50.019410 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:48:50.019386    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e4b174_0878_4db6_b106_02a6166a9fb8.slice/crio-8acc23adbb73d321f5538e0e68371d7f9d9c1ae670524d89dfb6a513629ccb34 WatchSource:0}: Error finding container 8acc23adbb73d321f5538e0e68371d7f9d9c1ae670524d89dfb6a513629ccb34: Status 404 returned error can't find the container with id 8acc23adbb73d321f5538e0e68371d7f9d9c1ae670524d89dfb6a513629ccb34
Jul 31 12:48:50.162933 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.162895    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5" event={"ID":"de11ce2a-0a8f-482c-9912-2c246d9591df","Type":"ContainerStarted","Data":"c783ca9a7947f22f287150c543772cc9be6fbabacb6f6e5ac9e0ec5f5f7432a3"}
Jul 31 12:48:50.163099 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.162958    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5" event={"ID":"de11ce2a-0a8f-482c-9912-2c246d9591df","Type":"ContainerStarted","Data":"039e27bc851fdfa12f2325b5db497802a76e864da81b13efe2562fe336a79998"}
Jul 31 12:48:50.163099 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.163064    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:48:50.164622 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.164599    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4" event={"ID":"441df164-4272-47c7-a81e-eec2e94c6344","Type":"ContainerStarted","Data":"1dd599652dfa9368d73c19047f6bb182bf6f7942f7abdd0d6e92d48653b0678a"}
Jul 31 12:48:50.164732 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.164639    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:48:50.164732 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.164662    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4" event={"ID":"441df164-4272-47c7-a81e-eec2e94c6344","Type":"ContainerStarted","Data":"11e7ebe48be8a227c2b61a4016a0fd8fecefeb662784ec296ea07830031519e1"}
Jul 31 12:48:50.166117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.166088    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w" event={"ID":"b4e4b174-0878-4db6-b106-02a6166a9fb8","Type":"ContainerStarted","Data":"ff24a99025e96e5a19a1aae231d15962f267505e6b1496a6e07ca4499920b7e3"}
Jul 31 12:48:50.166213 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.166136    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w" event={"ID":"b4e4b174-0878-4db6-b106-02a6166a9fb8","Type":"ContainerStarted","Data":"8acc23adbb73d321f5538e0e68371d7f9d9c1ae670524d89dfb6a513629ccb34"}
Jul 31 12:48:50.166213 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.166157    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:48:50.183647 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.183577    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5" podStartSLOduration=1.568339893 podStartE2EDuration="2.183564227s" totalImagesPullingTime="615.224334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:48:50.18145216 +0000 UTC m=+2018.168673700" watchObservedRunningTime="2026-07-31 12:48:50.183564227 +0000 UTC m=+2018.170785766"
Jul 31 12:48:50.201737 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.201701    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4" podStartSLOduration=1.201687619 podStartE2EDuration="1.201687619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:48:50.201418927 +0000 UTC m=+2018.188640467" watchObservedRunningTime="2026-07-31 12:48:50.201687619 +0000 UTC m=+2018.188909185"
Jul 31 12:48:50.220941 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:50.220897    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w" podStartSLOduration=1.220881013 podStartE2EDuration="1.220881013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:48:50.219607396 +0000 UTC m=+2018.206828950" watchObservedRunningTime="2026-07-31 12:48:50.220881013 +0000 UTC m=+2018.208102553"
Jul 31 12:48:56.176701 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:48:56.176676    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:49:01.173923 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:01.173893    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:49:01.174419 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:01.174303    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:49:01.228848 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:01.228814    2586 generic.go:356] "Generic (PLEG): container finished" podID="eae6299a-e317-4294-a257-b4ae9b353c86" containerID="1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66" exitCode=0
Jul 31 12:49:01.229043 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:01.228894    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" event={"ID":"eae6299a-e317-4294-a257-b4ae9b353c86","Type":"ContainerDied","Data":"1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66"}
Jul 31 12:49:01.229414 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:01.229392    2586 scope.go:117] "RemoveContainer" containerID="1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66"
Jul 31 12:49:01.230610 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:01.230570    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:49:01.230713 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:01.230658    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:49:01.230713 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:01.230704    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:49:02.147381 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:02.147313    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:49:02.237060 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:02.236478    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" event={"ID":"eae6299a-e317-4294-a257-b4ae9b353c86","Type":"ContainerStarted","Data":"9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc"}
Jul 31 12:49:02.787907 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:02.787875    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:49:02.788829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:02.788808    2586 patch_prober.go:28] interesting pod/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 31 12:49:02.788896 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:02.788856    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 31 12:49:03.787925 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:03.787889    2586 patch_prober.go:28] interesting pod/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 31 12:49:03.788373 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:03.787945    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 31 12:49:04.787915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:04.787878    2586 patch_prober.go:28] interesting pod/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 31 12:49:04.788098 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:04.787940    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 31 12:49:05.787606 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:05.787569    2586 patch_prober.go:28] interesting pod/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 31 12:49:05.787852 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:05.787632    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 31 12:49:06.788138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:06.788099    2586 patch_prober.go:28] interesting pod/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 31 12:49:06.788528 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:06.788155    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 31 12:49:07.791784 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:07.791755    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:49:07.792173 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:07.792010    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:49:07.792845 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:07.792829    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:49:19.088219 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.088182    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"]
Jul 31 12:49:19.088680 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.088440    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5" podUID="de11ce2a-0a8f-482c-9912-2c246d9591df" containerName="maas-api" containerID="cri-o://c783ca9a7947f22f287150c543772cc9be6fbabacb6f6e5ac9e0ec5f5f7432a3" gracePeriod=30
Jul 31 12:49:19.146342 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.146305    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"]
Jul 31 12:49:19.146586 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.146545    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w" podUID="b4e4b174-0878-4db6-b106-02a6166a9fb8" containerName="payload-processing" containerID="cri-o://ff24a99025e96e5a19a1aae231d15962f267505e6b1496a6e07ca4499920b7e3" gracePeriod=30
Jul 31 12:49:19.154104 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.154075    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"]
Jul 31 12:49:19.154376 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.154343    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4" podUID="441df164-4272-47c7-a81e-eec2e94c6344" containerName="payload-pre-processing" containerID="cri-o://1dd599652dfa9368d73c19047f6bb182bf6f7942f7abdd0d6e92d48653b0678a" gracePeriod=30
Jul 31 12:49:19.345535 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.345438    2586 generic.go:356] "Generic (PLEG): container finished" podID="de11ce2a-0a8f-482c-9912-2c246d9591df" containerID="c783ca9a7947f22f287150c543772cc9be6fbabacb6f6e5ac9e0ec5f5f7432a3" exitCode=0
Jul 31 12:49:19.345700 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.345536    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5" event={"ID":"de11ce2a-0a8f-482c-9912-2c246d9591df","Type":"ContainerDied","Data":"c783ca9a7947f22f287150c543772cc9be6fbabacb6f6e5ac9e0ec5f5f7432a3"}
Jul 31 12:49:19.347473 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.347370    2586 generic.go:356] "Generic (PLEG): container finished" podID="441df164-4272-47c7-a81e-eec2e94c6344" containerID="1dd599652dfa9368d73c19047f6bb182bf6f7942f7abdd0d6e92d48653b0678a" exitCode=0
Jul 31 12:49:19.347473 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.347466    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4" event={"ID":"441df164-4272-47c7-a81e-eec2e94c6344","Type":"ContainerDied","Data":"1dd599652dfa9368d73c19047f6bb182bf6f7942f7abdd0d6e92d48653b0678a"}
Jul 31 12:49:19.349517 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.349491    2586 generic.go:356] "Generic (PLEG): container finished" podID="b4e4b174-0878-4db6-b106-02a6166a9fb8" containerID="ff24a99025e96e5a19a1aae231d15962f267505e6b1496a6e07ca4499920b7e3" exitCode=0
Jul 31 12:49:19.349888 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.349605    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w" event={"ID":"b4e4b174-0878-4db6-b106-02a6166a9fb8","Type":"ContainerDied","Data":"ff24a99025e96e5a19a1aae231d15962f267505e6b1496a6e07ca4499920b7e3"}
Jul 31 12:49:19.424083 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.424058    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:49:19.426038 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.426021    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:49:19.431020 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.431000    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de11ce2a-0a8f-482c-9912-2c246d9591df-openshift-service-ca\") pod \"de11ce2a-0a8f-482c-9912-2c246d9591df\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") "
Jul 31 12:49:19.431100 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.431050    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp7zb\" (UniqueName: \"kubernetes.io/projected/b4e4b174-0878-4db6-b106-02a6166a9fb8-kube-api-access-qp7zb\") pod \"b4e4b174-0878-4db6-b106-02a6166a9fb8\" (UID: \"b4e4b174-0878-4db6-b106-02a6166a9fb8\") "
Jul 31 12:49:19.431197 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.431178    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4b174-0878-4db6-b106-02a6166a9fb8-plugins-config-volume\") pod \"b4e4b174-0878-4db6-b106-02a6166a9fb8\" (UID: \"b4e4b174-0878-4db6-b106-02a6166a9fb8\") "
Jul 31 12:49:19.431280 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.431250    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjpvq\" (UniqueName: \"kubernetes.io/projected/de11ce2a-0a8f-482c-9912-2c246d9591df-kube-api-access-vjpvq\") pod \"de11ce2a-0a8f-482c-9912-2c246d9591df\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") "
Jul 31 12:49:19.431346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.431296    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de11ce2a-0a8f-482c-9912-2c246d9591df-maas-api-tls\") pod \"de11ce2a-0a8f-482c-9912-2c246d9591df\" (UID: \"de11ce2a-0a8f-482c-9912-2c246d9591df\") "
Jul 31 12:49:19.431403 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.431349    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de11ce2a-0a8f-482c-9912-2c246d9591df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de11ce2a-0a8f-482c-9912-2c246d9591df" (UID: "de11ce2a-0a8f-482c-9912-2c246d9591df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:49:19.431650 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.431602    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de11ce2a-0a8f-482c-9912-2c246d9591df-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:19.431650 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.431616    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e4b174-0878-4db6-b106-02a6166a9fb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4e4b174-0878-4db6-b106-02a6166a9fb8" (UID: "b4e4b174-0878-4db6-b106-02a6166a9fb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:49:19.433985 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.433959    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de11ce2a-0a8f-482c-9912-2c246d9591df-kube-api-access-vjpvq" (OuterVolumeSpecName: "kube-api-access-vjpvq") pod "de11ce2a-0a8f-482c-9912-2c246d9591df" (UID: "de11ce2a-0a8f-482c-9912-2c246d9591df"). InnerVolumeSpecName "kube-api-access-vjpvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:49:19.434095 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.434067    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de11ce2a-0a8f-482c-9912-2c246d9591df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de11ce2a-0a8f-482c-9912-2c246d9591df" (UID: "de11ce2a-0a8f-482c-9912-2c246d9591df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:49:19.434189 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.434172    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e4b174-0878-4db6-b106-02a6166a9fb8-kube-api-access-qp7zb" (OuterVolumeSpecName: "kube-api-access-qp7zb") pod "b4e4b174-0878-4db6-b106-02a6166a9fb8" (UID: "b4e4b174-0878-4db6-b106-02a6166a9fb8"). InnerVolumeSpecName "kube-api-access-qp7zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:49:19.455962 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.455927    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:49:19.532804 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.532773    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2zrq\" (UniqueName: \"kubernetes.io/projected/441df164-4272-47c7-a81e-eec2e94c6344-kube-api-access-j2zrq\") pod \"441df164-4272-47c7-a81e-eec2e94c6344\" (UID: \"441df164-4272-47c7-a81e-eec2e94c6344\") "
Jul 31 12:49:19.532978 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.532858    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441df164-4272-47c7-a81e-eec2e94c6344-plugins-config-volume\") pod \"441df164-4272-47c7-a81e-eec2e94c6344\" (UID: \"441df164-4272-47c7-a81e-eec2e94c6344\") "
Jul 31 12:49:19.533097 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.533080    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7zb\" (UniqueName: \"kubernetes.io/projected/b4e4b174-0878-4db6-b106-02a6166a9fb8-kube-api-access-qp7zb\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:19.533163 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.533103    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4b174-0878-4db6-b106-02a6166a9fb8-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:19.533163 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.533118    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjpvq\" (UniqueName: \"kubernetes.io/projected/de11ce2a-0a8f-482c-9912-2c246d9591df-kube-api-access-vjpvq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:19.533163 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.533132    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de11ce2a-0a8f-482c-9912-2c246d9591df-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:19.533359 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.533236    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441df164-4272-47c7-a81e-eec2e94c6344-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "441df164-4272-47c7-a81e-eec2e94c6344" (UID: "441df164-4272-47c7-a81e-eec2e94c6344"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:49:19.534846 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.534825    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441df164-4272-47c7-a81e-eec2e94c6344-kube-api-access-j2zrq" (OuterVolumeSpecName: "kube-api-access-j2zrq") pod "441df164-4272-47c7-a81e-eec2e94c6344" (UID: "441df164-4272-47c7-a81e-eec2e94c6344"). InnerVolumeSpecName "kube-api-access-j2zrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:49:19.634393 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.634329    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441df164-4272-47c7-a81e-eec2e94c6344-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:19.634393 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:19.634352    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2zrq\" (UniqueName: \"kubernetes.io/projected/441df164-4272-47c7-a81e-eec2e94c6344-kube-api-access-j2zrq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:20.355575 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.355485    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5" event={"ID":"de11ce2a-0a8f-482c-9912-2c246d9591df","Type":"ContainerDied","Data":"039e27bc851fdfa12f2325b5db497802a76e864da81b13efe2562fe336a79998"}
Jul 31 12:49:20.355575 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.355566    2586 scope.go:117] "RemoveContainer" containerID="c783ca9a7947f22f287150c543772cc9be6fbabacb6f6e5ac9e0ec5f5f7432a3"
Jul 31 12:49:20.355575 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.355495    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"
Jul 31 12:49:20.357057 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.356901    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"
Jul 31 12:49:20.357057 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.356929    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4" event={"ID":"441df164-4272-47c7-a81e-eec2e94c6344","Type":"ContainerDied","Data":"11e7ebe48be8a227c2b61a4016a0fd8fecefeb662784ec296ea07830031519e1"}
Jul 31 12:49:20.358448 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.358419    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w" event={"ID":"b4e4b174-0878-4db6-b106-02a6166a9fb8","Type":"ContainerDied","Data":"8acc23adbb73d321f5538e0e68371d7f9d9c1ae670524d89dfb6a513629ccb34"}
Jul 31 12:49:20.358448 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.358433    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"
Jul 31 12:49:20.365499 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.365482    2586 scope.go:117] "RemoveContainer" containerID="1dd599652dfa9368d73c19047f6bb182bf6f7942f7abdd0d6e92d48653b0678a"
Jul 31 12:49:20.374359 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.374340    2586 scope.go:117] "RemoveContainer" containerID="ff24a99025e96e5a19a1aae231d15962f267505e6b1496a6e07ca4499920b7e3"
Jul 31 12:49:20.378829 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.378810    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"]
Jul 31 12:49:20.382486 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.382470    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fed071b-85d474fd74-46zl5"]
Jul 31 12:49:20.392498 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.392468    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"]
Jul 31 12:49:20.395070 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.395052    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fed071b-6dc8494bf6-cq8d4"]
Jul 31 12:49:20.404953 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.404936    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"]
Jul 31 12:49:20.408638 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.408616    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fed071b-6ccb9fb44b-b9c9w"]
Jul 31 12:49:20.486120 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.486091    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441df164-4272-47c7-a81e-eec2e94c6344" path="/var/lib/kubelet/pods/441df164-4272-47c7-a81e-eec2e94c6344/volumes"
Jul 31 12:49:20.486625 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.486601    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e4b174-0878-4db6-b106-02a6166a9fb8" path="/var/lib/kubelet/pods/b4e4b174-0878-4db6-b106-02a6166a9fb8/volumes"
Jul 31 12:49:20.487080 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.487061    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de11ce2a-0a8f-482c-9912-2c246d9591df" path="/var/lib/kubelet/pods/de11ce2a-0a8f-482c-9912-2c246d9591df/volumes"
Jul 31 12:49:20.508450 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.508427    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"]
Jul 31 12:49:20.508705 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:20.508680    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy" containerID="cri-o://9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc" gracePeriod=30
Jul 31 12:49:22.795380 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:22.795347    2586 patch_prober.go:28] interesting pod/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 12:49:22.795772 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:22.795409    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 12:49:25.308360 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.308325    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"]
Jul 31 12:49:25.308911 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.308895    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de11ce2a-0a8f-482c-9912-2c246d9591df" containerName="maas-api"
Jul 31 12:49:25.308959 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.308914    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="de11ce2a-0a8f-482c-9912-2c246d9591df" containerName="maas-api"
Jul 31 12:49:25.308959 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.308928    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e4b174-0878-4db6-b106-02a6166a9fb8" containerName="payload-processing"
Jul 31 12:49:25.308959 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.308937    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e4b174-0878-4db6-b106-02a6166a9fb8" containerName="payload-processing"
Jul 31 12:49:25.309062 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.308963    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441df164-4272-47c7-a81e-eec2e94c6344" containerName="payload-pre-processing"
Jul 31 12:49:25.309062 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.308973    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="441df164-4272-47c7-a81e-eec2e94c6344" containerName="payload-pre-processing"
Jul 31 12:49:25.309125 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.309067    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e4b174-0878-4db6-b106-02a6166a9fb8" containerName="payload-processing"
Jul 31 12:49:25.309125 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.309083    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="de11ce2a-0a8f-482c-9912-2c246d9591df" containerName="maas-api"
Jul 31 12:49:25.309125 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.309092    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="441df164-4272-47c7-a81e-eec2e94c6344" containerName="payload-pre-processing"
Jul 31 12:49:25.314106 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.314081    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.316435 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.316412    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-46d34f7d-serving-cert\""
Jul 31 12:49:25.322345 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.322322    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"]
Jul 31 12:49:25.383704 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.383670    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6729e01-eeb3-412d-8b4a-232b60d1e790-openshift-service-ca\") pod \"maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.383889 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.383734    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6729e01-eeb3-412d-8b4a-232b60d1e790-maas-api-tls\") pod \"maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.383889 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.383767    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdjpm\" (UniqueName: \"kubernetes.io/projected/d6729e01-eeb3-412d-8b4a-232b60d1e790-kube-api-access-qdjpm\") pod \"maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.484978 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.484946    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6729e01-eeb3-412d-8b4a-232b60d1e790-maas-api-tls\") pod \"maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.485196 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.484994    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdjpm\" (UniqueName: \"kubernetes.io/projected/d6729e01-eeb3-412d-8b4a-232b60d1e790-kube-api-access-qdjpm\") pod \"maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.485196 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.485129    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6729e01-eeb3-412d-8b4a-232b60d1e790-openshift-service-ca\") pod \"maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.485841 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.485817    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6729e01-eeb3-412d-8b4a-232b60d1e790-openshift-service-ca\") pod \"maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.487791 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.487770    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6729e01-eeb3-412d-8b4a-232b60d1e790-maas-api-tls\") pod \"maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.500404 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.500380    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdjpm\" (UniqueName: \"kubernetes.io/projected/d6729e01-eeb3-412d-8b4a-232b60d1e790-kube-api-access-qdjpm\") pod \"maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.627414 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.627378    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:25.769229 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.769200    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:49:25.789010 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.788985    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae6299a-e317-4294-a257-b4ae9b353c86-istio-podinfo\") pod \"eae6299a-e317-4294-a257-b4ae9b353c86\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") "
Jul 31 12:49:25.789154 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789090    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-socket\") pod \"eae6299a-e317-4294-a257-b4ae9b353c86\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") "
Jul 31 12:49:25.789154 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789121    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-certs\") pod \"eae6299a-e317-4294-a257-b4ae9b353c86\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") "
Jul 31 12:49:25.789154 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789150    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae6299a-e317-4294-a257-b4ae9b353c86-istiod-ca-cert\") pod \"eae6299a-e317-4294-a257-b4ae9b353c86\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") "
Jul 31 12:49:25.789346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789189    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-credential-socket\") pod \"eae6299a-e317-4294-a257-b4ae9b353c86\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") "
Jul 31 12:49:25.789346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789229    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-istio-token\") pod \"eae6299a-e317-4294-a257-b4ae9b353c86\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") "
Jul 31 12:49:25.789346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789253    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l89h\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-kube-api-access-8l89h\") pod \"eae6299a-e317-4294-a257-b4ae9b353c86\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") "
Jul 31 12:49:25.789346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789304    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-envoy\") pod \"eae6299a-e317-4294-a257-b4ae9b353c86\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") "
Jul 31 12:49:25.789346 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789323    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-data\") pod \"eae6299a-e317-4294-a257-b4ae9b353c86\" (UID: \"eae6299a-e317-4294-a257-b4ae9b353c86\") "
Jul 31 12:49:25.789590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789382    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eae6299a-e317-4294-a257-b4ae9b353c86" (UID: "eae6299a-e317-4294-a257-b4ae9b353c86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:49:25.789590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789440    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eae6299a-e317-4294-a257-b4ae9b353c86" (UID: "eae6299a-e317-4294-a257-b4ae9b353c86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:49:25.789590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789544    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae6299a-e317-4294-a257-b4ae9b353c86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eae6299a-e317-4294-a257-b4ae9b353c86" (UID: "eae6299a-e317-4294-a257-b4ae9b353c86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:49:25.789748 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789633    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eae6299a-e317-4294-a257-b4ae9b353c86" (UID: "eae6299a-e317-4294-a257-b4ae9b353c86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:49:25.789808 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789778    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:25.789808 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789800    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:25.789898 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789814    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae6299a-e317-4294-a257-b4ae9b353c86-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:25.789898 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789827    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:25.789898 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.789834    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-data" (OuterVolumeSpecName: "istio-data") pod "eae6299a-e317-4294-a257-b4ae9b353c86" (UID: "eae6299a-e317-4294-a257-b4ae9b353c86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:49:25.791121 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.791091    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eae6299a-e317-4294-a257-b4ae9b353c86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eae6299a-e317-4294-a257-b4ae9b353c86" (UID: "eae6299a-e317-4294-a257-b4ae9b353c86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 12:49:25.791500 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.791469    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-kube-api-access-8l89h" (OuterVolumeSpecName: "kube-api-access-8l89h") pod "eae6299a-e317-4294-a257-b4ae9b353c86" (UID: "eae6299a-e317-4294-a257-b4ae9b353c86"). InnerVolumeSpecName "kube-api-access-8l89h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:49:25.791855 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.791831    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eae6299a-e317-4294-a257-b4ae9b353c86" (UID: "eae6299a-e317-4294-a257-b4ae9b353c86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:49:25.791855 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.791844    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-istio-token" (OuterVolumeSpecName: "istio-token") pod "eae6299a-e317-4294-a257-b4ae9b353c86" (UID: "eae6299a-e317-4294-a257-b4ae9b353c86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:49:25.890724 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.890651    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae6299a-e317-4294-a257-b4ae9b353c86-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:25.890724 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.890677    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:25.890724 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.890686    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l89h\" (UniqueName: \"kubernetes.io/projected/eae6299a-e317-4294-a257-b4ae9b353c86-kube-api-access-8l89h\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:25.890724 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.890696    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:25.890724 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.890705    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae6299a-e317-4294-a257-b4ae9b353c86-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:25.902572 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.902546    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"]
Jul 31 12:49:25.902909 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.902897    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy"
Jul 31 12:49:25.902966 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.902910    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy"
Jul 31 12:49:25.903011 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.902976    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy"
Jul 31 12:49:25.906197 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.906181    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:25.908651 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.908637    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-46d34f7d\""
Jul 31 12:49:25.908651 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.908648    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-46d34f7d-dockercfg-dmbc2\""
Jul 31 12:49:25.918576 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.918556    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"]
Jul 31 12:49:25.936996 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.936976    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"]
Jul 31 12:49:25.937497 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.937482    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy"
Jul 31 12:49:25.937556 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.937500    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy"
Jul 31 12:49:25.937592 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.937572    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" containerName="istio-proxy"
Jul 31 12:49:25.940354 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.940339    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:25.951435 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.951412    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"]
Jul 31 12:49:25.976037 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.975923    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"]
Jul 31 12:49:25.978132 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:49:25.978102    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6729e01_eeb3_412d_8b4a_232b60d1e790.slice/crio-a4734ebc09cf40dd49af9ff106224d0f440a3edbbfa3922fe4133aa97e1ea604 WatchSource:0}: Error finding container a4734ebc09cf40dd49af9ff106224d0f440a3edbbfa3922fe4133aa97e1ea604: Status 404 returned error can't find the container with id a4734ebc09cf40dd49af9ff106224d0f440a3edbbfa3922fe4133aa97e1ea604
Jul 31 12:49:25.991607 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.991586    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4sd\" (UniqueName: \"kubernetes.io/projected/56eb4c57-fe24-49f9-85d5-accdd0a5b274-kube-api-access-vq4sd\") pod \"payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr\" (UID: \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\") " pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:25.991687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.991617    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5drgc\" (UniqueName: \"kubernetes.io/projected/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-kube-api-access-5drgc\") pod \"payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n\" (UID: \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:25.991687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.991635    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb4c57-fe24-49f9-85d5-accdd0a5b274-plugins-config-volume\") pod \"payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr\" (UID: \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\") " pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:25.991687 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:25.991663    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n\" (UID: \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:26.092982 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.092950    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4sd\" (UniqueName: \"kubernetes.io/projected/56eb4c57-fe24-49f9-85d5-accdd0a5b274-kube-api-access-vq4sd\") pod \"payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr\" (UID: \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\") " pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:26.093117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.092985    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drgc\" (UniqueName: \"kubernetes.io/projected/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-kube-api-access-5drgc\") pod \"payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n\" (UID: \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:26.093117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.093015    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb4c57-fe24-49f9-85d5-accdd0a5b274-plugins-config-volume\") pod \"payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr\" (UID: \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\") " pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:26.093117 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.093049    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n\" (UID: \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:26.093981 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.093955    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n\" (UID: \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:26.094086 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.093955    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb4c57-fe24-49f9-85d5-accdd0a5b274-plugins-config-volume\") pod \"payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr\" (UID: \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\") " pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:26.100798 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.100773    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4sd\" (UniqueName: \"kubernetes.io/projected/56eb4c57-fe24-49f9-85d5-accdd0a5b274-kube-api-access-vq4sd\") pod \"payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr\" (UID: \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\") " pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:26.100882 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.100834    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5drgc\" (UniqueName: \"kubernetes.io/projected/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-kube-api-access-5drgc\") pod \"payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n\" (UID: \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:26.216399 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.216366    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:26.252956 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.252930    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:26.362282 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.362245    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"]
Jul 31 12:49:26.363683 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:49:26.363651    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d34d9dc_ea1c_4589_a4a0_ef6a1c9fca3b.slice/crio-1d1e718b71846e9209aba2217af2e1b604b31c55af72a9226f333204cccbe40a WatchSource:0}: Error finding container 1d1e718b71846e9209aba2217af2e1b604b31c55af72a9226f333204cccbe40a: Status 404 returned error can't find the container with id 1d1e718b71846e9209aba2217af2e1b604b31c55af72a9226f333204cccbe40a
Jul 31 12:49:26.393340 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.393306    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n" event={"ID":"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b","Type":"ContainerStarted","Data":"1d1e718b71846e9209aba2217af2e1b604b31c55af72a9226f333204cccbe40a"}
Jul 31 12:49:26.395859 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.395827    2586 generic.go:356] "Generic (PLEG): container finished" podID="eae6299a-e317-4294-a257-b4ae9b353c86" containerID="9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc" exitCode=0
Jul 31 12:49:26.395982 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.395880    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"
Jul 31 12:49:26.395982 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.395944    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" event={"ID":"eae6299a-e317-4294-a257-b4ae9b353c86","Type":"ContainerDied","Data":"9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc"}
Jul 31 12:49:26.396099 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.395991    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk" event={"ID":"eae6299a-e317-4294-a257-b4ae9b353c86","Type":"ContainerDied","Data":"ebbb90bab5cbaac4159cd847bfb6f58fa8f2501f3b42f95be8bcf6f0a5bb1ee4"}
Jul 31 12:49:26.396099 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.396023    2586 scope.go:117] "RemoveContainer" containerID="9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc"
Jul 31 12:49:26.399915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.399892    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf" event={"ID":"d6729e01-eeb3-412d-8b4a-232b60d1e790","Type":"ContainerStarted","Data":"a4734ebc09cf40dd49af9ff106224d0f440a3edbbfa3922fe4133aa97e1ea604"}
Jul 31 12:49:26.408009 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.407978    2586 scope.go:117] "RemoveContainer" containerID="1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66"
Jul 31 12:49:26.409553 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.409338    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"]
Jul 31 12:49:26.412925 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:49:26.412899    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56eb4c57_fe24_49f9_85d5_accdd0a5b274.slice/crio-a56943c7e355680cdc811234b564245d7f210c2351832f9824dd9975f5132eb7 WatchSource:0}: Error finding container a56943c7e355680cdc811234b564245d7f210c2351832f9824dd9975f5132eb7: Status 404 returned error can't find the container with id a56943c7e355680cdc811234b564245d7f210c2351832f9824dd9975f5132eb7
Jul 31 12:49:26.418900 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.418879    2586 scope.go:117] "RemoveContainer" containerID="9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc"
Jul 31 12:49:26.419188 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:49:26.419160    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc\": container with ID starting with 9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc not found: ID does not exist" containerID="9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc"
Jul 31 12:49:26.419310 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.419201    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc"} err="failed to get container status \"9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc\": rpc error: code = NotFound desc = could not find container \"9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc\": container with ID starting with 9ea72cac00cc713ba91dfcb16b35e4c63119a96c2262ccf578aef54d2cc68dcc not found: ID does not exist"
Jul 31 12:49:26.419310 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.419226    2586 scope.go:117] "RemoveContainer" containerID="1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66"
Jul 31 12:49:26.419600 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:49:26.419575    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66\": container with ID starting with 1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66 not found: ID does not exist" containerID="1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66"
Jul 31 12:49:26.419666 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.419612    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66"} err="failed to get container status \"1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66\": rpc error: code = NotFound desc = could not find container \"1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66\": container with ID starting with 1d00f029598e5e2bf41c5ac705a093163fa6bc12b7b92854efd83e9774be4f66 not found: ID does not exist"
Jul 31 12:49:26.425845 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.425824    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"]
Jul 31 12:49:26.432367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.432346    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8fed071b-openshift-default-5b9c568d74-zxzqk"]
Jul 31 12:49:26.484676 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:26.484643    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae6299a-e317-4294-a257-b4ae9b353c86" path="/var/lib/kubelet/pods/eae6299a-e317-4294-a257-b4ae9b353c86/volumes"
Jul 31 12:49:27.407080 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.407051    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr" event={"ID":"56eb4c57-fe24-49f9-85d5-accdd0a5b274","Type":"ContainerStarted","Data":"c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c"}
Jul 31 12:49:27.407525 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.407101    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr" event={"ID":"56eb4c57-fe24-49f9-85d5-accdd0a5b274","Type":"ContainerStarted","Data":"a56943c7e355680cdc811234b564245d7f210c2351832f9824dd9975f5132eb7"}
Jul 31 12:49:27.407525 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.407125    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:27.408630 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.408606    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n" event={"ID":"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b","Type":"ContainerStarted","Data":"d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f"}
Jul 31 12:49:27.408678 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.408669    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:27.410851 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.410826    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf" event={"ID":"d6729e01-eeb3-412d-8b4a-232b60d1e790","Type":"ContainerStarted","Data":"7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b"}
Jul 31 12:49:27.410966 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.410934    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:27.436955 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.436907    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr" podStartSLOduration=2.436892969 podStartE2EDuration="2.436892969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:49:27.422393584 +0000 UTC m=+2055.409615124" watchObservedRunningTime="2026-07-31 12:49:27.436892969 +0000 UTC m=+2055.424114509"
Jul 31 12:49:27.438204 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.438166    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n" podStartSLOduration=2.438157167 podStartE2EDuration="2.438157167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:49:27.436879818 +0000 UTC m=+2055.424101360" watchObservedRunningTime="2026-07-31 12:49:27.438157167 +0000 UTC m=+2055.425378706"
Jul 31 12:49:27.454621 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:27.454575    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf" podStartSLOduration=2.07291866 podStartE2EDuration="2.454562172s" totalImagesPullingTime="381.643512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:49:27.452439494 +0000 UTC m=+2055.439661044" watchObservedRunningTime="2026-07-31 12:49:27.454562172 +0000 UTC m=+2055.441783712"
Jul 31 12:49:33.421448 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:33.421415    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:38.418003 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:38.417970    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:38.418403 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:38.418047    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:55.774189 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:55.774158    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"]
Jul 31 12:49:55.774736 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:55.774394    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf" podUID="d6729e01-eeb3-412d-8b4a-232b60d1e790" containerName="maas-api" containerID="cri-o://7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b" gracePeriod=30
Jul 31 12:49:55.813219 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:55.813188    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"]
Jul 31 12:49:55.813438 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:55.813412    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr" podUID="56eb4c57-fe24-49f9-85d5-accdd0a5b274" containerName="payload-processing" containerID="cri-o://c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c" gracePeriod=30
Jul 31 12:49:55.820317 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:55.820286    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"]
Jul 31 12:49:55.820586 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:55.820529    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n" podUID="5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b" containerName="payload-pre-processing" containerID="cri-o://d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f" gracePeriod=30
Jul 31 12:49:56.111728 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.111701    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:56.123890 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.123865    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:56.141343 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.141323    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:56.254696 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.254669    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6729e01-eeb3-412d-8b4a-232b60d1e790-maas-api-tls\") pod \"d6729e01-eeb3-412d-8b4a-232b60d1e790\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") "
Jul 31 12:49:56.254872 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.254715    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdjpm\" (UniqueName: \"kubernetes.io/projected/d6729e01-eeb3-412d-8b4a-232b60d1e790-kube-api-access-qdjpm\") pod \"d6729e01-eeb3-412d-8b4a-232b60d1e790\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") "
Jul 31 12:49:56.254872 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.254759    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq4sd\" (UniqueName: \"kubernetes.io/projected/56eb4c57-fe24-49f9-85d5-accdd0a5b274-kube-api-access-vq4sd\") pod \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\" (UID: \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\") "
Jul 31 12:49:56.254872 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.254805    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5drgc\" (UniqueName: \"kubernetes.io/projected/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-kube-api-access-5drgc\") pod \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\" (UID: \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\") "
Jul 31 12:49:56.254872 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.254834    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-plugins-config-volume\") pod \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\" (UID: \"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b\") "
Jul 31 12:49:56.255590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.254915    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb4c57-fe24-49f9-85d5-accdd0a5b274-plugins-config-volume\") pod \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\" (UID: \"56eb4c57-fe24-49f9-85d5-accdd0a5b274\") "
Jul 31 12:49:56.255590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.254991    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6729e01-eeb3-412d-8b4a-232b60d1e790-openshift-service-ca\") pod \"d6729e01-eeb3-412d-8b4a-232b60d1e790\" (UID: \"d6729e01-eeb3-412d-8b4a-232b60d1e790\") "
Jul 31 12:49:56.255590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.255474    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b" (UID: "5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:49:56.255590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.255474    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56eb4c57-fe24-49f9-85d5-accdd0a5b274-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56eb4c57-fe24-49f9-85d5-accdd0a5b274" (UID: "56eb4c57-fe24-49f9-85d5-accdd0a5b274"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:49:56.255590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.255545    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6729e01-eeb3-412d-8b4a-232b60d1e790-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6729e01-eeb3-412d-8b4a-232b60d1e790" (UID: "d6729e01-eeb3-412d-8b4a-232b60d1e790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:49:56.256981 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.256961    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6729e01-eeb3-412d-8b4a-232b60d1e790-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6729e01-eeb3-412d-8b4a-232b60d1e790" (UID: "d6729e01-eeb3-412d-8b4a-232b60d1e790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:49:56.256981 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.256964    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-kube-api-access-5drgc" (OuterVolumeSpecName: "kube-api-access-5drgc") pod "5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b" (UID: "5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b"). InnerVolumeSpecName "kube-api-access-5drgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:49:56.257216 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.257010    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56eb4c57-fe24-49f9-85d5-accdd0a5b274-kube-api-access-vq4sd" (OuterVolumeSpecName: "kube-api-access-vq4sd") pod "56eb4c57-fe24-49f9-85d5-accdd0a5b274" (UID: "56eb4c57-fe24-49f9-85d5-accdd0a5b274"). InnerVolumeSpecName "kube-api-access-vq4sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:49:56.257216 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.257032    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6729e01-eeb3-412d-8b4a-232b60d1e790-kube-api-access-qdjpm" (OuterVolumeSpecName: "kube-api-access-qdjpm") pod "d6729e01-eeb3-412d-8b4a-232b60d1e790" (UID: "d6729e01-eeb3-412d-8b4a-232b60d1e790"). InnerVolumeSpecName "kube-api-access-qdjpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:49:56.356429 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.356342    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:56.356429 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.356372    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56eb4c57-fe24-49f9-85d5-accdd0a5b274-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:56.356429 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.356384    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6729e01-eeb3-412d-8b4a-232b60d1e790-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:56.356429 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.356398    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6729e01-eeb3-412d-8b4a-232b60d1e790-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:56.356429 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.356410    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdjpm\" (UniqueName: \"kubernetes.io/projected/d6729e01-eeb3-412d-8b4a-232b60d1e790-kube-api-access-qdjpm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:56.356429 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.356423    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4sd\" (UniqueName: \"kubernetes.io/projected/56eb4c57-fe24-49f9-85d5-accdd0a5b274-kube-api-access-vq4sd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:56.356723 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.356435    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5drgc\" (UniqueName: \"kubernetes.io/projected/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b-kube-api-access-5drgc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:49:56.553241 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.553212    2586 generic.go:356] "Generic (PLEG): container finished" podID="56eb4c57-fe24-49f9-85d5-accdd0a5b274" containerID="c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c" exitCode=0
Jul 31 12:49:56.553371 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.553259    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"
Jul 31 12:49:56.553371 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.553298    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr" event={"ID":"56eb4c57-fe24-49f9-85d5-accdd0a5b274","Type":"ContainerDied","Data":"c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c"}
Jul 31 12:49:56.553371 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.553352    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr" event={"ID":"56eb4c57-fe24-49f9-85d5-accdd0a5b274","Type":"ContainerDied","Data":"a56943c7e355680cdc811234b564245d7f210c2351832f9824dd9975f5132eb7"}
Jul 31 12:49:56.553511 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.553374    2586 scope.go:117] "RemoveContainer" containerID="c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c"
Jul 31 12:49:56.554568 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.554537    2586 generic.go:356] "Generic (PLEG): container finished" podID="5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b" containerID="d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f" exitCode=0
Jul 31 12:49:56.554661 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.554587    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"
Jul 31 12:49:56.554726 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.554588    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n" event={"ID":"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b","Type":"ContainerDied","Data":"d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f"}
Jul 31 12:49:56.554788 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.554751    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n" event={"ID":"5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b","Type":"ContainerDied","Data":"1d1e718b71846e9209aba2217af2e1b604b31c55af72a9226f333204cccbe40a"}
Jul 31 12:49:56.556078 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.556060    2586 generic.go:356] "Generic (PLEG): container finished" podID="d6729e01-eeb3-412d-8b4a-232b60d1e790" containerID="7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b" exitCode=0
Jul 31 12:49:56.556176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.556095    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf" event={"ID":"d6729e01-eeb3-412d-8b4a-232b60d1e790","Type":"ContainerDied","Data":"7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b"}
Jul 31 12:49:56.556176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.556117    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"
Jul 31 12:49:56.556290 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.556124    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf" event={"ID":"d6729e01-eeb3-412d-8b4a-232b60d1e790","Type":"ContainerDied","Data":"a4734ebc09cf40dd49af9ff106224d0f440a3edbbfa3922fe4133aa97e1ea604"}
Jul 31 12:49:56.562902 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.562886    2586 scope.go:117] "RemoveContainer" containerID="c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c"
Jul 31 12:49:56.563140 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:49:56.563123    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c\": container with ID starting with c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c not found: ID does not exist" containerID="c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c"
Jul 31 12:49:56.563194 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.563146    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c"} err="failed to get container status \"c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c\": rpc error: code = NotFound desc = could not find container \"c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c\": container with ID starting with c8f77b3dcd493707ead3dc7d83361d5515801fc30e4fe4353ea6b959a9bf826c not found: ID does not exist"
Jul 31 12:49:56.563194 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.563160    2586 scope.go:117] "RemoveContainer" containerID="d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f"
Jul 31 12:49:56.570572 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.570557    2586 scope.go:117] "RemoveContainer" containerID="d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f"
Jul 31 12:49:56.570796 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:49:56.570781    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f\": container with ID starting with d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f not found: ID does not exist" containerID="d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f"
Jul 31 12:49:56.570842 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.570802    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f"} err="failed to get container status \"d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f\": rpc error: code = NotFound desc = could not find container \"d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f\": container with ID starting with d0976c9315662d9abc21b220d0f1b5ba898c975326f1fa7cd0718a83dd098a3f not found: ID does not exist"
Jul 31 12:49:56.570842 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.570817    2586 scope.go:117] "RemoveContainer" containerID="7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b"
Jul 31 12:49:56.576540 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.576521    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"]
Jul 31 12:49:56.580144 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.580110    2586 scope.go:117] "RemoveContainer" containerID="7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b"
Jul 31 12:49:56.580867 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:49:56.580842    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b\": container with ID starting with 7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b not found: ID does not exist" containerID="7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b"
Jul 31 12:49:56.580933 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.580875    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b"} err="failed to get container status \"7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b\": rpc error: code = NotFound desc = could not find container \"7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b\": container with ID starting with 7c7b793f44046397ba3a93f02759e8cb199ebf9ef9fd654381f386346f09284b not found: ID does not exist"
Jul 31 12:49:56.581972 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.581956    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46d34f7d-548bfcc54b-slvvf"]
Jul 31 12:49:56.591421 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.591399    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"]
Jul 31 12:49:56.595646 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.595629    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46d34f7d-85985f74d-clt4n"]
Jul 31 12:49:56.606988 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.606934    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"]
Jul 31 12:49:56.610777 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:56.610758    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46d34f7d-fc666fcd6-lwpwr"]
Jul 31 12:49:58.484719 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:58.484687    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56eb4c57-fe24-49f9-85d5-accdd0a5b274" path="/var/lib/kubelet/pods/56eb4c57-fe24-49f9-85d5-accdd0a5b274/volumes"
Jul 31 12:49:58.485106 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:58.485042    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b" path="/var/lib/kubelet/pods/5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b/volumes"
Jul 31 12:49:58.485394 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:49:58.485381    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6729e01-eeb3-412d-8b4a-232b60d1e790" path="/var/lib/kubelet/pods/d6729e01-eeb3-412d-8b4a-232b60d1e790/volumes"
Jul 31 12:51:52.262692 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.262660    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"]
Jul 31 12:51:52.263138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.263034    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56eb4c57-fe24-49f9-85d5-accdd0a5b274" containerName="payload-processing"
Jul 31 12:51:52.263138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.263046    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="56eb4c57-fe24-49f9-85d5-accdd0a5b274" containerName="payload-processing"
Jul 31 12:51:52.263138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.263067    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b" containerName="payload-pre-processing"
Jul 31 12:51:52.263138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.263072    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b" containerName="payload-pre-processing"
Jul 31 12:51:52.263138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.263082    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6729e01-eeb3-412d-8b4a-232b60d1e790" containerName="maas-api"
Jul 31 12:51:52.263138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.263087    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6729e01-eeb3-412d-8b4a-232b60d1e790" containerName="maas-api"
Jul 31 12:51:52.263138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.263137    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="56eb4c57-fe24-49f9-85d5-accdd0a5b274" containerName="payload-processing"
Jul 31 12:51:52.263385 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.263146    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d34d9dc-ea1c-4589-a4a0-ef6a1c9fca3b" containerName="payload-pre-processing"
Jul 31 12:51:52.263385 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.263153    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6729e01-eeb3-412d-8b4a-232b60d1e790" containerName="maas-api"
Jul 31 12:51:52.265755 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.265740    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.268020 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.268001    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-88cd166c-openshift-default-dockercfg-nfzlk\""
Jul 31 12:51:52.285471 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.285447    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"]
Jul 31 12:51:52.385605 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.385565    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-certs\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.385772 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.385613    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-podinfo\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.385772 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.385646    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbkk\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-kube-api-access-dpbkk\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.385772 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.385692    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-data\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.385772 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.385720    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c028d7f-c3da-4772-8562-dfdebea7283d-istiod-ca-cert\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.385772 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.385746    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-token\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.386023 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.385786    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-socket\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.386023 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.385859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-credential-socket\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.386023 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.385903    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-envoy\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487197 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487171    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-credential-socket\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487197 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487201    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-envoy\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487446 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487274    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-certs\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487446 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487302    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-podinfo\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487446 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487328    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbkk\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-kube-api-access-dpbkk\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487611 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487541    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-data\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487676 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487608    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-credential-socket\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487676 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487654    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-certs\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487733    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c028d7f-c3da-4772-8562-dfdebea7283d-istiod-ca-cert\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487816 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487769    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-token\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487855 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487824    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-socket\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.487855 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.487837    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-data\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.488164 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.488146    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-socket\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.488421 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.488402    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c028d7f-c3da-4772-8562-dfdebea7283d-istiod-ca-cert\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.490122 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.490091    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-envoy\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.490223 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.490213    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-podinfo\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.495078 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.495054    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-token\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.495456 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.495435    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbkk\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-kube-api-access-dpbkk\") pod \"e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") " pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.576429 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.576342    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:52.713213 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.713189    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"]
Jul 31 12:51:52.715132 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:51:52.715108    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c028d7f_c3da_4772_8562_dfdebea7283d.slice/crio-b07cb30cc7ad5ec44ec9678e879b90d829f1989a7e31c81409c392732f2759a9 WatchSource:0}: Error finding container b07cb30cc7ad5ec44ec9678e879b90d829f1989a7e31c81409c392732f2759a9: Status 404 returned error can't find the container with id b07cb30cc7ad5ec44ec9678e879b90d829f1989a7e31c81409c392732f2759a9
Jul 31 12:51:52.717786 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.717751    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:51:52.717864 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.717844    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:51:52.717906 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:52.717891    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:51:53.111578 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:53.111545    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" event={"ID":"9c028d7f-c3da-4772-8562-dfdebea7283d","Type":"ContainerStarted","Data":"e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac"}
Jul 31 12:51:53.111765 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:53.111603    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" event={"ID":"9c028d7f-c3da-4772-8562-dfdebea7283d","Type":"ContainerStarted","Data":"b07cb30cc7ad5ec44ec9678e879b90d829f1989a7e31c81409c392732f2759a9"}
Jul 31 12:51:53.131050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:53.131000    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" podStartSLOduration=1.130985167 podStartE2EDuration="1.130985167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:51:53.128808094 +0000 UTC m=+2201.116029633" watchObservedRunningTime="2026-07-31 12:51:53.130985167 +0000 UTC m=+2201.118206706"
Jul 31 12:51:53.577335 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:53.577298    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:53.582128 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:53.582104    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:54.116046 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:54.116016    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:54.117026 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:54.117008    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:51:56.131382 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.131347    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"]
Jul 31 12:51:56.135131 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.135110    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.137498 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.137477    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-88cd166c-serving-cert\""
Jul 31 12:51:56.145290 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.145254    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"]
Jul 31 12:51:56.219894 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.219860    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f6f121-f279-4c37-aed3-dd3957035a2c-openshift-service-ca\") pod \"maas-api-e2e-mt-88cd166c-6996d6c497-b5czb\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.220142 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.219908    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f6f121-f279-4c37-aed3-dd3957035a2c-maas-api-tls\") pod \"maas-api-e2e-mt-88cd166c-6996d6c497-b5czb\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.220142 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.219936    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxn4h\" (UniqueName: \"kubernetes.io/projected/e4f6f121-f279-4c37-aed3-dd3957035a2c-kube-api-access-jxn4h\") pod \"maas-api-e2e-mt-88cd166c-6996d6c497-b5czb\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.321557 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.321523    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f6f121-f279-4c37-aed3-dd3957035a2c-openshift-service-ca\") pod \"maas-api-e2e-mt-88cd166c-6996d6c497-b5czb\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.321557 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.321560    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f6f121-f279-4c37-aed3-dd3957035a2c-maas-api-tls\") pod \"maas-api-e2e-mt-88cd166c-6996d6c497-b5czb\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.321756 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.321580    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxn4h\" (UniqueName: \"kubernetes.io/projected/e4f6f121-f279-4c37-aed3-dd3957035a2c-kube-api-access-jxn4h\") pod \"maas-api-e2e-mt-88cd166c-6996d6c497-b5czb\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.322220 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.322198    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f6f121-f279-4c37-aed3-dd3957035a2c-openshift-service-ca\") pod \"maas-api-e2e-mt-88cd166c-6996d6c497-b5czb\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.323909 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.323890    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f6f121-f279-4c37-aed3-dd3957035a2c-maas-api-tls\") pod \"maas-api-e2e-mt-88cd166c-6996d6c497-b5czb\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.329883 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.329860    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxn4h\" (UniqueName: \"kubernetes.io/projected/e4f6f121-f279-4c37-aed3-dd3957035a2c-kube-api-access-jxn4h\") pod \"maas-api-e2e-mt-88cd166c-6996d6c497-b5czb\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.451281 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.451242    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:56.593358 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.593323    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"]
Jul 31 12:51:56.595652 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:51:56.595630    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f6f121_f279_4c37_aed3_dd3957035a2c.slice/crio-e9c6aebaea24eca76897d13988262e3d457c376d540f632376b01ff106ad43b7 WatchSource:0}: Error finding container e9c6aebaea24eca76897d13988262e3d457c376d540f632376b01ff106ad43b7: Status 404 returned error can't find the container with id e9c6aebaea24eca76897d13988262e3d457c376d540f632376b01ff106ad43b7
Jul 31 12:51:56.759126 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.759038    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"]
Jul 31 12:51:56.763728 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.763705    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:51:56.770780 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.770754    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-88cd166c\""
Jul 31 12:51:56.770902 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.770764    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-88cd166c-dockercfg-659r5\""
Jul 31 12:51:56.776487 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.776441    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"]
Jul 31 12:51:56.786934 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.786914    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"]
Jul 31 12:51:56.790334 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.790319    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:51:56.804446 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.804428    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"]
Jul 31 12:51:56.826666 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.826646    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv2rq\" (UniqueName: \"kubernetes.io/projected/3e6fade7-6774-4bd1-912b-03d07ca84806-kube-api-access-sv2rq\") pod \"payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp\" (UID: \"3e6fade7-6774-4bd1-912b-03d07ca84806\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:51:56.826768 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.826689    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6fade7-6774-4bd1-912b-03d07ca84806-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp\" (UID: \"3e6fade7-6774-4bd1-912b-03d07ca84806\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:51:56.928175 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.928145    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-plugins-config-volume\") pod \"payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn\" (UID: \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\") " pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:51:56.928367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.928195    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv2rq\" (UniqueName: \"kubernetes.io/projected/3e6fade7-6774-4bd1-912b-03d07ca84806-kube-api-access-sv2rq\") pod \"payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp\" (UID: \"3e6fade7-6774-4bd1-912b-03d07ca84806\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:51:56.928367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.928216    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q98mt\" (UniqueName: \"kubernetes.io/projected/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-kube-api-access-q98mt\") pod \"payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn\" (UID: \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\") " pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:51:56.928367 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.928323    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6fade7-6774-4bd1-912b-03d07ca84806-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp\" (UID: \"3e6fade7-6774-4bd1-912b-03d07ca84806\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:51:56.928989 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.928961    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6fade7-6774-4bd1-912b-03d07ca84806-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp\" (UID: \"3e6fade7-6774-4bd1-912b-03d07ca84806\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:51:56.936146 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:56.936125    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv2rq\" (UniqueName: \"kubernetes.io/projected/3e6fade7-6774-4bd1-912b-03d07ca84806-kube-api-access-sv2rq\") pod \"payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp\" (UID: \"3e6fade7-6774-4bd1-912b-03d07ca84806\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:51:57.029210 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:57.029123    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-plugins-config-volume\") pod \"payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn\" (UID: \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\") " pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:51:57.029210 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:57.029198    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q98mt\" (UniqueName: \"kubernetes.io/projected/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-kube-api-access-q98mt\") pod \"payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn\" (UID: \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\") " pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:51:57.029786 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:57.029768    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-plugins-config-volume\") pod \"payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn\" (UID: \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\") " pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:51:57.036634 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:57.036603    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q98mt\" (UniqueName: \"kubernetes.io/projected/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-kube-api-access-q98mt\") pod \"payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn\" (UID: \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\") " pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:51:57.074049 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:57.074026    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:51:57.100085 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:57.100060    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:51:57.136164 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:57.136127    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb" event={"ID":"e4f6f121-f279-4c37-aed3-dd3957035a2c","Type":"ContainerStarted","Data":"e9c6aebaea24eca76897d13988262e3d457c376d540f632376b01ff106ad43b7"}
Jul 31 12:51:57.221244 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:57.221207    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"]
Jul 31 12:51:57.224537 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:51:57.224493    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6fade7_6774_4bd1_912b_03d07ca84806.slice/crio-ec9e2c4223c19c289f88b0f17c053419d9e90228e06404a42dca41753a68101b WatchSource:0}: Error finding container ec9e2c4223c19c289f88b0f17c053419d9e90228e06404a42dca41753a68101b: Status 404 returned error can't find the container with id ec9e2c4223c19c289f88b0f17c053419d9e90228e06404a42dca41753a68101b
Jul 31 12:51:57.249017 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:57.248987    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"]
Jul 31 12:51:57.265420 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:51:57.265394    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4207f6c1_1033_4d78_ad73_5cc6d6fb7b4c.slice/crio-d06a533e3afcd1c8db8978f163bdd6077ed13c77c00d35b74093364746eef075 WatchSource:0}: Error finding container d06a533e3afcd1c8db8978f163bdd6077ed13c77c00d35b74093364746eef075: Status 404 returned error can't find the container with id d06a533e3afcd1c8db8978f163bdd6077ed13c77c00d35b74093364746eef075
Jul 31 12:51:58.144078 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.144041    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb" event={"ID":"e4f6f121-f279-4c37-aed3-dd3957035a2c","Type":"ContainerStarted","Data":"bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74"}
Jul 31 12:51:58.144557 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.144147    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:51:58.145688 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.145664    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn" event={"ID":"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c","Type":"ContainerStarted","Data":"572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93"}
Jul 31 12:51:58.145806 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.145713    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:51:58.145806 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.145739    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn" event={"ID":"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c","Type":"ContainerStarted","Data":"d06a533e3afcd1c8db8978f163bdd6077ed13c77c00d35b74093364746eef075"}
Jul 31 12:51:58.147486 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.147464    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp" event={"ID":"3e6fade7-6774-4bd1-912b-03d07ca84806","Type":"ContainerStarted","Data":"9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519"}
Jul 31 12:51:58.147603 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.147501    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp" event={"ID":"3e6fade7-6774-4bd1-912b-03d07ca84806","Type":"ContainerStarted","Data":"ec9e2c4223c19c289f88b0f17c053419d9e90228e06404a42dca41753a68101b"}
Jul 31 12:51:58.147603 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.147597    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:51:58.159959 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.159915    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb" podStartSLOduration=1.3755422 podStartE2EDuration="2.15990013s" totalImagesPullingTime="784.35793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:51:58.159559721 +0000 UTC m=+2206.146781262" watchObservedRunningTime="2026-07-31 12:51:58.15990013 +0000 UTC m=+2206.147121670"
Jul 31 12:51:58.177471 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.177422    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn" podStartSLOduration=2.177402604 podStartE2EDuration="2.177402604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:51:58.174923691 +0000 UTC m=+2206.162145232" watchObservedRunningTime="2026-07-31 12:51:58.177402604 +0000 UTC m=+2206.164624148"
Jul 31 12:51:58.191552 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:51:58.191462    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp" podStartSLOduration=2.191449387 podStartE2EDuration="2.191449387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:51:58.189747679 +0000 UTC m=+2206.176969220" watchObservedRunningTime="2026-07-31 12:51:58.191449387 +0000 UTC m=+2206.178670927"
Jul 31 12:52:04.166309 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:04.166255    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:52:09.118112 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.118072    2586 patch_prober.go:28] interesting pod/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 31 12:52:09.118539 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.118142    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 31 12:52:09.154889 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.154848    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp" podUID="3e6fade7-6774-4bd1-912b-03d07ca84806" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.63:9004: connect: connection refused"
Jul 31 12:52:09.155466 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.155448    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:52:09.209728 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.209700    2586 generic.go:356] "Generic (PLEG): container finished" podID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerID="e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac" exitCode=0
Jul 31 12:52:09.209898 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.209752    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" event={"ID":"9c028d7f-c3da-4772-8562-dfdebea7283d","Type":"ContainerDied","Data":"e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac"}
Jul 31 12:52:09.210175 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.210158    2586 scope.go:117] "RemoveContainer" containerID="e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac"
Jul 31 12:52:09.211237 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.211205    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:52:09.211329 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.211309    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:52:09.211373 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:09.211348    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:52:10.216436 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:10.216400    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" event={"ID":"9c028d7f-c3da-4772-8562-dfdebea7283d","Type":"ContainerStarted","Data":"3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25"}
Jul 31 12:52:10.576785 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:10.576697    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:52:10.576931 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:10.576884    2586 patch_prober.go:28] interesting pod/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 31 12:52:10.576974 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:10.576942    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 31 12:52:11.577310 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:11.577277    2586 patch_prober.go:28] interesting pod/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 31 12:52:11.577712 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:11.577325    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 31 12:52:12.577554 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:12.577524    2586 patch_prober.go:28] interesting pod/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 31 12:52:12.577990 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:12.577578    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 31 12:52:13.577415 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:13.577379    2586 patch_prober.go:28] interesting pod/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 31 12:52:13.577599 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:13.577438    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 31 12:52:14.577287 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:14.577226    2586 patch_prober.go:28] interesting pod/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 31 12:52:14.577494 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:14.577320    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 31 12:52:15.581389 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:15.581356    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:52:15.581818 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:15.581698    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:52:15.582426 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:15.582409    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:52:19.154659 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:19.154630    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:52:37.459619 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.459536    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"]
Jul 31 12:52:37.460138 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.459753    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb" podUID="e4f6f121-f279-4c37-aed3-dd3957035a2c" containerName="maas-api" containerID="cri-o://bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74" gracePeriod=30
Jul 31 12:52:37.514006 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.513976    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"]
Jul 31 12:52:37.514232 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.514208    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn" podUID="4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c" containerName="payload-processing" containerID="cri-o://572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93" gracePeriod=30
Jul 31 12:52:37.518855 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.518692    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"]
Jul 31 12:52:37.518960 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.518887    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp" podUID="3e6fade7-6774-4bd1-912b-03d07ca84806" containerName="payload-pre-processing" containerID="cri-o://9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519" gracePeriod=30
Jul 31 12:52:37.792424 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.792400    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:52:37.800199 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.799532    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:52:37.827699 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.827673    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:52:37.906487 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.906453    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv2rq\" (UniqueName: \"kubernetes.io/projected/3e6fade7-6774-4bd1-912b-03d07ca84806-kube-api-access-sv2rq\") pod \"3e6fade7-6774-4bd1-912b-03d07ca84806\" (UID: \"3e6fade7-6774-4bd1-912b-03d07ca84806\") "
Jul 31 12:52:37.906668 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.906495    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-plugins-config-volume\") pod \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\" (UID: \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\") "
Jul 31 12:52:37.906668 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.906535    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6fade7-6774-4bd1-912b-03d07ca84806-plugins-config-volume\") pod \"3e6fade7-6774-4bd1-912b-03d07ca84806\" (UID: \"3e6fade7-6774-4bd1-912b-03d07ca84806\") "
Jul 31 12:52:37.906668 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.906566    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q98mt\" (UniqueName: \"kubernetes.io/projected/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-kube-api-access-q98mt\") pod \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\" (UID: \"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c\") "
Jul 31 12:52:37.906668 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.906610    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f6f121-f279-4c37-aed3-dd3957035a2c-maas-api-tls\") pod \"e4f6f121-f279-4c37-aed3-dd3957035a2c\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") "
Jul 31 12:52:37.906668 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.906650    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f6f121-f279-4c37-aed3-dd3957035a2c-openshift-service-ca\") pod \"e4f6f121-f279-4c37-aed3-dd3957035a2c\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") "
Jul 31 12:52:37.906922 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.906692    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxn4h\" (UniqueName: \"kubernetes.io/projected/e4f6f121-f279-4c37-aed3-dd3957035a2c-kube-api-access-jxn4h\") pod \"e4f6f121-f279-4c37-aed3-dd3957035a2c\" (UID: \"e4f6f121-f279-4c37-aed3-dd3957035a2c\") "
Jul 31 12:52:37.907066 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.907019    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c" (UID: "4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:52:37.907066 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.907044    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f6f121-f279-4c37-aed3-dd3957035a2c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4f6f121-f279-4c37-aed3-dd3957035a2c" (UID: "e4f6f121-f279-4c37-aed3-dd3957035a2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:52:37.907066 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.907043    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6fade7-6774-4bd1-912b-03d07ca84806-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e6fade7-6774-4bd1-912b-03d07ca84806" (UID: "3e6fade7-6774-4bd1-912b-03d07ca84806"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:52:37.908670 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.908647    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-kube-api-access-q98mt" (OuterVolumeSpecName: "kube-api-access-q98mt") pod "4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c" (UID: "4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c"). InnerVolumeSpecName "kube-api-access-q98mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:52:37.908798 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.908777    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f6f121-f279-4c37-aed3-dd3957035a2c-kube-api-access-jxn4h" (OuterVolumeSpecName: "kube-api-access-jxn4h") pod "e4f6f121-f279-4c37-aed3-dd3957035a2c" (UID: "e4f6f121-f279-4c37-aed3-dd3957035a2c"). InnerVolumeSpecName "kube-api-access-jxn4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:52:37.909042 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.909018    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6fade7-6774-4bd1-912b-03d07ca84806-kube-api-access-sv2rq" (OuterVolumeSpecName: "kube-api-access-sv2rq") pod "3e6fade7-6774-4bd1-912b-03d07ca84806" (UID: "3e6fade7-6774-4bd1-912b-03d07ca84806"). InnerVolumeSpecName "kube-api-access-sv2rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:52:37.909042 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:37.909019    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f6f121-f279-4c37-aed3-dd3957035a2c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4f6f121-f279-4c37-aed3-dd3957035a2c" (UID: "e4f6f121-f279-4c37-aed3-dd3957035a2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:52:38.008050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.007984    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f6f121-f279-4c37-aed3-dd3957035a2c-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:38.008050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.008008    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f6f121-f279-4c37-aed3-dd3957035a2c-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:38.008050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.008019    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxn4h\" (UniqueName: \"kubernetes.io/projected/e4f6f121-f279-4c37-aed3-dd3957035a2c-kube-api-access-jxn4h\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:38.008050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.008028    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv2rq\" (UniqueName: \"kubernetes.io/projected/3e6fade7-6774-4bd1-912b-03d07ca84806-kube-api-access-sv2rq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:38.008050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.008038    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:38.008050 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.008047    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6fade7-6774-4bd1-912b-03d07ca84806-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:38.008351 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.008055    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q98mt\" (UniqueName: \"kubernetes.io/projected/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c-kube-api-access-q98mt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:38.349113 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.349022    2586 generic.go:356] "Generic (PLEG): container finished" podID="e4f6f121-f279-4c37-aed3-dd3957035a2c" containerID="bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74" exitCode=0
Jul 31 12:52:38.349113 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.349071    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"
Jul 31 12:52:38.349113 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.349100    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb" event={"ID":"e4f6f121-f279-4c37-aed3-dd3957035a2c","Type":"ContainerDied","Data":"bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74"}
Jul 31 12:52:38.349396 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.349148    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb" event={"ID":"e4f6f121-f279-4c37-aed3-dd3957035a2c","Type":"ContainerDied","Data":"e9c6aebaea24eca76897d13988262e3d457c376d540f632376b01ff106ad43b7"}
Jul 31 12:52:38.349396 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.349168    2586 scope.go:117] "RemoveContainer" containerID="bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74"
Jul 31 12:52:38.350426 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.350403    2586 generic.go:356] "Generic (PLEG): container finished" podID="4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c" containerID="572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93" exitCode=0
Jul 31 12:52:38.350544 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.350433    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn" event={"ID":"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c","Type":"ContainerDied","Data":"572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93"}
Jul 31 12:52:38.350544 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.350455    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"
Jul 31 12:52:38.350544 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.350479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn" event={"ID":"4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c","Type":"ContainerDied","Data":"d06a533e3afcd1c8db8978f163bdd6077ed13c77c00d35b74093364746eef075"}
Jul 31 12:52:38.351848 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.351825    2586 generic.go:356] "Generic (PLEG): container finished" podID="3e6fade7-6774-4bd1-912b-03d07ca84806" containerID="9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519" exitCode=0
Jul 31 12:52:38.351965 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.351864    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"
Jul 31 12:52:38.351965 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.351907    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp" event={"ID":"3e6fade7-6774-4bd1-912b-03d07ca84806","Type":"ContainerDied","Data":"9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519"}
Jul 31 12:52:38.351965 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.351961    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp" event={"ID":"3e6fade7-6774-4bd1-912b-03d07ca84806","Type":"ContainerDied","Data":"ec9e2c4223c19c289f88b0f17c053419d9e90228e06404a42dca41753a68101b"}
Jul 31 12:52:38.360627 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.360598    2586 scope.go:117] "RemoveContainer" containerID="bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74"
Jul 31 12:52:38.361104 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:52:38.361080    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74\": container with ID starting with bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74 not found: ID does not exist" containerID="bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74"
Jul 31 12:52:38.361250 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.361225    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74"} err="failed to get container status \"bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74\": rpc error: code = NotFound desc = could not find container \"bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74\": container with ID starting with bf82540f98f77f980547bcd4144561c20700403a91acbdb0fca44e874df22c74 not found: ID does not exist"
Jul 31 12:52:38.361456 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.361441    2586 scope.go:117] "RemoveContainer" containerID="572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93"
Jul 31 12:52:38.372680 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.372663    2586 scope.go:117] "RemoveContainer" containerID="572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93"
Jul 31 12:52:38.372914 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:52:38.372895    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93\": container with ID starting with 572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93 not found: ID does not exist" containerID="572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93"
Jul 31 12:52:38.372958 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.372921    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93"} err="failed to get container status \"572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93\": rpc error: code = NotFound desc = could not find container \"572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93\": container with ID starting with 572c5e5bd2abb3c44d95ab5a9238b1bfdac1c7ed473b05712b6acbe4ba11ca93 not found: ID does not exist"
Jul 31 12:52:38.372958 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.372937    2586 scope.go:117] "RemoveContainer" containerID="9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519"
Jul 31 12:52:38.380112 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.380095    2586 scope.go:117] "RemoveContainer" containerID="9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519"
Jul 31 12:52:38.380344 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:52:38.380328    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519\": container with ID starting with 9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519 not found: ID does not exist" containerID="9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519"
Jul 31 12:52:38.380415 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.380348    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519"} err="failed to get container status \"9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519\": rpc error: code = NotFound desc = could not find container \"9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519\": container with ID starting with 9eacfee70817c5ca631d4bcad4118e062cae217d89d4765d0bc06b75ed644519 not found: ID does not exist"
Jul 31 12:52:38.387535 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.387514    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"]
Jul 31 12:52:38.393076 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.393056    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88cd166c-6996d6c497-b5czb"]
Jul 31 12:52:38.402202 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.402175    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"]
Jul 31 12:52:38.405516 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.405497    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88cd166c-7fc49f8f6-cfghp"]
Jul 31 12:52:38.414702 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.414684    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"]
Jul 31 12:52:38.418122 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.418105    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88cd166c-84b84fb74-tsfqn"]
Jul 31 12:52:38.485058 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.485033    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e6fade7-6774-4bd1-912b-03d07ca84806" path="/var/lib/kubelet/pods/3e6fade7-6774-4bd1-912b-03d07ca84806/volumes"
Jul 31 12:52:38.485392 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.485379    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c" path="/var/lib/kubelet/pods/4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c/volumes"
Jul 31 12:52:38.485711 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:38.485700    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4f6f121-f279-4c37-aed3-dd3957035a2c" path="/var/lib/kubelet/pods/e4f6f121-f279-4c37-aed3-dd3957035a2c/volumes"
Jul 31 12:52:43.846394 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:43.846359    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"]
Jul 31 12:52:43.846877 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:43.846563    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy" containerID="cri-o://3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25" gracePeriod=30
Jul 31 12:52:45.584239 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:45.584207    2586 patch_prober.go:28] interesting pod/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 12:52:45.584600 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:45.584289    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 12:52:49.113490 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.113432    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:52:49.203387 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203356    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpbkk\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-kube-api-access-dpbkk\") pod \"9c028d7f-c3da-4772-8562-dfdebea7283d\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") "
Jul 31 12:52:49.203537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203401    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-credential-socket\") pod \"9c028d7f-c3da-4772-8562-dfdebea7283d\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") "
Jul 31 12:52:49.203537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203432    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-token\") pod \"9c028d7f-c3da-4772-8562-dfdebea7283d\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") "
Jul 31 12:52:49.203537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203471    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c028d7f-c3da-4772-8562-dfdebea7283d-istiod-ca-cert\") pod \"9c028d7f-c3da-4772-8562-dfdebea7283d\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") "
Jul 31 12:52:49.203537 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203494    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-socket\") pod \"9c028d7f-c3da-4772-8562-dfdebea7283d\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") "
Jul 31 12:52:49.203753 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203676    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-envoy\") pod \"9c028d7f-c3da-4772-8562-dfdebea7283d\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") "
Jul 31 12:52:49.203753 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203722    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-data\") pod \"9c028d7f-c3da-4772-8562-dfdebea7283d\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") "
Jul 31 12:52:49.203864 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203767    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-podinfo\") pod \"9c028d7f-c3da-4772-8562-dfdebea7283d\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") "
Jul 31 12:52:49.203864 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203709    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c028d7f-c3da-4772-8562-dfdebea7283d" (UID: "9c028d7f-c3da-4772-8562-dfdebea7283d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:52:49.203864 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203799    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-certs\") pod \"9c028d7f-c3da-4772-8562-dfdebea7283d\" (UID: \"9c028d7f-c3da-4772-8562-dfdebea7283d\") "
Jul 31 12:52:49.203864 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203777    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c028d7f-c3da-4772-8562-dfdebea7283d" (UID: "9c028d7f-c3da-4772-8562-dfdebea7283d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:52:49.204075 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203948    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c028d7f-c3da-4772-8562-dfdebea7283d" (UID: "9c028d7f-c3da-4772-8562-dfdebea7283d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:52:49.204075 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.203991    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c028d7f-c3da-4772-8562-dfdebea7283d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c028d7f-c3da-4772-8562-dfdebea7283d" (UID: "9c028d7f-c3da-4772-8562-dfdebea7283d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:52:49.204246 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.204135    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c028d7f-c3da-4772-8562-dfdebea7283d" (UID: "9c028d7f-c3da-4772-8562-dfdebea7283d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:52:49.204246 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.204144    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:49.204246 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.204163    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:49.204246 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.204191    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c028d7f-c3da-4772-8562-dfdebea7283d-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:49.204246 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.204200    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:49.205555 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.205531    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c028d7f-c3da-4772-8562-dfdebea7283d" (UID: "9c028d7f-c3da-4772-8562-dfdebea7283d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:52:49.205784 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.205767    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c028d7f-c3da-4772-8562-dfdebea7283d" (UID: "9c028d7f-c3da-4772-8562-dfdebea7283d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 12:52:49.205935 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.205906    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c028d7f-c3da-4772-8562-dfdebea7283d" (UID: "9c028d7f-c3da-4772-8562-dfdebea7283d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:52:49.205935 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.205918    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-kube-api-access-dpbkk" (OuterVolumeSpecName: "kube-api-access-dpbkk") pod "9c028d7f-c3da-4772-8562-dfdebea7283d" (UID: "9c028d7f-c3da-4772-8562-dfdebea7283d"). InnerVolumeSpecName "kube-api-access-dpbkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:52:49.305085 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.305058    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:49.305085 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.305080    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:49.305237 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.305089    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpbkk\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-kube-api-access-dpbkk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:49.305237 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.305099    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:49.305237 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.305109    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c028d7f-c3da-4772-8562-dfdebea7283d-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:52:49.409602 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.409573    2586 generic.go:356] "Generic (PLEG): container finished" podID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerID="3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25" exitCode=0
Jul 31 12:52:49.409748 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.409630    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"
Jul 31 12:52:49.409748 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.409662    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" event={"ID":"9c028d7f-c3da-4772-8562-dfdebea7283d","Type":"ContainerDied","Data":"3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25"}
Jul 31 12:52:49.409748 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.409742    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t" event={"ID":"9c028d7f-c3da-4772-8562-dfdebea7283d","Type":"ContainerDied","Data":"b07cb30cc7ad5ec44ec9678e879b90d829f1989a7e31c81409c392732f2759a9"}
Jul 31 12:52:49.409850 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.409767    2586 scope.go:117] "RemoveContainer" containerID="3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25"
Jul 31 12:52:49.418635 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.418619    2586 scope.go:117] "RemoveContainer" containerID="e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac"
Jul 31 12:52:49.426564 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.426545    2586 scope.go:117] "RemoveContainer" containerID="3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25"
Jul 31 12:52:49.426798 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:52:49.426779    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25\": container with ID starting with 3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25 not found: ID does not exist" containerID="3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25"
Jul 31 12:52:49.426842 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.426806    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25"} err="failed to get container status \"3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25\": rpc error: code = NotFound desc = could not find container \"3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25\": container with ID starting with 3e3cc3c06f3b64c604e661a687323190d5ca79026974bc585843e06bd78e9f25 not found: ID does not exist"
Jul 31 12:52:49.426842 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.426822    2586 scope.go:117] "RemoveContainer" containerID="e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac"
Jul 31 12:52:49.427083 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:52:49.427068    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac\": container with ID starting with e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac not found: ID does not exist" containerID="e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac"
Jul 31 12:52:49.427135 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.427086    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac"} err="failed to get container status \"e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac\": rpc error: code = NotFound desc = could not find container \"e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac\": container with ID starting with e69b70d6d6e9cb526a05c9b5eb6fb5118f5c66930f7594c2401c01bc055d75ac not found: ID does not exist"
Jul 31 12:52:49.430433 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.430413    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"]
Jul 31 12:52:49.433914 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:49.433895    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-88cd166c-openshift-default-7c66c8d4b4-hgf4t"]
Jul 31 12:52:50.484460 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:52:50.484430    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" path="/var/lib/kubelet/pods/9c028d7f-c3da-4772-8562-dfdebea7283d/volumes"
Jul 31 12:54:59.524468 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524434    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"]
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524907    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy"
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524922    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy"
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524935    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e6fade7-6774-4bd1-912b-03d07ca84806" containerName="payload-pre-processing"
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524945    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e6fade7-6774-4bd1-912b-03d07ca84806" containerName="payload-pre-processing"
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524954    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy"
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524961    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy"
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524972    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c" containerName="payload-processing"
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524977    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c" containerName="payload-processing"
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524984    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f6f121-f279-4c37-aed3-dd3957035a2c" containerName="maas-api"
Jul 31 12:54:59.525021 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.524989    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f6f121-f279-4c37-aed3-dd3957035a2c" containerName="maas-api"
Jul 31 12:54:59.525358 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.525054    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy"
Jul 31 12:54:59.525358 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.525062    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f6f121-f279-4c37-aed3-dd3957035a2c" containerName="maas-api"
Jul 31 12:54:59.525358 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.525069    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e6fade7-6774-4bd1-912b-03d07ca84806" containerName="payload-pre-processing"
Jul 31 12:54:59.525358 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.525075    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4207f6c1-1033-4d78-ad73-5cc6d6fb7b4c" containerName="payload-processing"
Jul 31 12:54:59.528078 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.528063    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.530627 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.530606    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bcf3ab8b-openshift-default-dockercfg-lnmkx\""
Jul 31 12:54:59.539454 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.539424    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"]
Jul 31 12:54:59.720708 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.720673    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-credential-socket\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.720885 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.720727    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-token\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.720885 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.720748    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-envoy\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.720885 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.720801    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlnk8\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-kube-api-access-qlnk8\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.720885 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.720841    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-socket\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.720885 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.720860    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f914c43b-d069-46fb-88bf-5c8c3472d596-istiod-ca-cert\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.721097 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.720903    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-podinfo\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.721097 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.720956    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-certs\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.721097 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.720984    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-data\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.821667 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821586    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-socket\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.821667 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821625    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f914c43b-d069-46fb-88bf-5c8c3472d596-istiod-ca-cert\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.821886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821669    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-podinfo\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.821886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821713    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-certs\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.821886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821746    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-data\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.821886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821783    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-credential-socket\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.821886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821846    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-token\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.821886 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821882    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-envoy\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.822190 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821906    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnk8\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-kube-api-access-qlnk8\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.822190 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.821985    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-socket\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.822190 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.822155    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-credential-socket\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.822366 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.822202    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-data\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.822366 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.822288    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-certs\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.822449 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.822432    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f914c43b-d069-46fb-88bf-5c8c3472d596-istiod-ca-cert\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.824066 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.824044    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-envoy\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.824260 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.824246    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-podinfo\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.830035 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.830011    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnk8\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-kube-api-access-qlnk8\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.830134 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.830109    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-token\") pod \"e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") " pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.839199 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.839183    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:54:59.972602 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.972574    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"]
Jul 31 12:54:59.974552 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:54:59.974529    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf914c43b_d069_46fb_88bf_5c8c3472d596.slice/crio-5de9acc916cb73ce947dde7f840a35882f0db5f2e2481b99bed9946f3ecab6a0 WatchSource:0}: Error finding container 5de9acc916cb73ce947dde7f840a35882f0db5f2e2481b99bed9946f3ecab6a0: Status 404 returned error can't find the container with id 5de9acc916cb73ce947dde7f840a35882f0db5f2e2481b99bed9946f3ecab6a0
Jul 31 12:54:59.976731 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.976693    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:54:59.976811 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.976782    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:54:59.976853 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:54:59.976829    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:55:00.013673 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:00.013646    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" event={"ID":"f914c43b-d069-46fb-88bf-5c8c3472d596","Type":"ContainerStarted","Data":"5de9acc916cb73ce947dde7f840a35882f0db5f2e2481b99bed9946f3ecab6a0"}
Jul 31 12:55:01.019611 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:01.019572    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" event={"ID":"f914c43b-d069-46fb-88bf-5c8c3472d596","Type":"ContainerStarted","Data":"ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d"}
Jul 31 12:55:01.041561 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:01.041517    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" podStartSLOduration=2.041501504 podStartE2EDuration="2.041501504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:55:01.039713365 +0000 UTC m=+2389.026934902" watchObservedRunningTime="2026-07-31 12:55:01.041501504 +0000 UTC m=+2389.028723044"
Jul 31 12:55:01.839454 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:01.839420    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:55:01.844249 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:01.844226    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:55:02.024209 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:02.024178    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:55:02.025217 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:02.025194    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:55:04.057415 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.057380    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"]
Jul 31 12:55:04.058833 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.058798    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c028d7f-c3da-4772-8562-dfdebea7283d" containerName="istio-proxy"
Jul 31 12:55:04.064540 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.064511    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.075550 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.071768    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bcf3ab8b-serving-cert\""
Jul 31 12:55:04.075550 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.074820    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"]
Jul 31 12:55:04.169074 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.169038    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42917d5-6221-466b-a7a1-e3dc58168bf6-maas-api-tls\") pod \"maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.169074 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.169086    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftpp\" (UniqueName: \"kubernetes.io/projected/e42917d5-6221-466b-a7a1-e3dc58168bf6-kube-api-access-sftpp\") pod \"maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.169319 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.169188    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42917d5-6221-466b-a7a1-e3dc58168bf6-openshift-service-ca\") pod \"maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.270590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.270554    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42917d5-6221-466b-a7a1-e3dc58168bf6-maas-api-tls\") pod \"maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.270590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.270596    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftpp\" (UniqueName: \"kubernetes.io/projected/e42917d5-6221-466b-a7a1-e3dc58168bf6-kube-api-access-sftpp\") pod \"maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.270891 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.270634    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42917d5-6221-466b-a7a1-e3dc58168bf6-openshift-service-ca\") pod \"maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.271253 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.271231    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42917d5-6221-466b-a7a1-e3dc58168bf6-openshift-service-ca\") pod \"maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.273161 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.273135    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42917d5-6221-466b-a7a1-e3dc58168bf6-maas-api-tls\") pod \"maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.279487 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.279459    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftpp\" (UniqueName: \"kubernetes.io/projected/e42917d5-6221-466b-a7a1-e3dc58168bf6-kube-api-access-sftpp\") pod \"maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.385026 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.384940    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:04.571582 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:55:04.571554    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42917d5_6221_466b_a7a1_e3dc58168bf6.slice/crio-07143199cd51f6bf890288a32d16f0cb852dc885a1429e047a4aa74a4dabbad0 WatchSource:0}: Error finding container 07143199cd51f6bf890288a32d16f0cb852dc885a1429e047a4aa74a4dabbad0: Status 404 returned error can't find the container with id 07143199cd51f6bf890288a32d16f0cb852dc885a1429e047a4aa74a4dabbad0
Jul 31 12:55:04.571714 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.571599    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"]
Jul 31 12:55:04.573008 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.572991    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 12:55:04.811142 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.811106    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"]
Jul 31 12:55:04.815758 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.815740    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:04.818488 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.818468    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bcf3ab8b-dockercfg-rmx8p\""
Jul 31 12:55:04.818622 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.818522    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bcf3ab8b\""
Jul 31 12:55:04.827518 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.827494    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"]
Jul 31 12:55:04.860611 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.860577    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"]
Jul 31 12:55:04.864116 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.864102    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:04.875551 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.875523    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk64x\" (UniqueName: \"kubernetes.io/projected/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-kube-api-access-rk64x\") pod \"payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9\" (UID: \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\") " pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:04.875686 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.875591    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-plugins-config-volume\") pod \"payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9\" (UID: \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\") " pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:04.877696 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.877671    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"]
Jul 31 12:55:04.976838 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.976803    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-plugins-config-volume\") pod \"payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9\" (UID: \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\") " pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:04.977041 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.976868    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003334c6-907c-4352-969d-f04b25dc7391-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd\" (UID: \"003334c6-907c-4352-969d-f04b25dc7391\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:04.977041 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.976888    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qtx\" (UniqueName: \"kubernetes.io/projected/003334c6-907c-4352-969d-f04b25dc7391-kube-api-access-n8qtx\") pod \"payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd\" (UID: \"003334c6-907c-4352-969d-f04b25dc7391\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:04.977041 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.976946    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk64x\" (UniqueName: \"kubernetes.io/projected/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-kube-api-access-rk64x\") pod \"payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9\" (UID: \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\") " pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:04.977665 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.977647    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-plugins-config-volume\") pod \"payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9\" (UID: \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\") " pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:04.985284 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:04.985242    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk64x\" (UniqueName: \"kubernetes.io/projected/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-kube-api-access-rk64x\") pod \"payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9\" (UID: \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\") " pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:05.054558 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:05.054455    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x" event={"ID":"e42917d5-6221-466b-a7a1-e3dc58168bf6","Type":"ContainerStarted","Data":"07143199cd51f6bf890288a32d16f0cb852dc885a1429e047a4aa74a4dabbad0"}
Jul 31 12:55:05.077590 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:05.077566    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003334c6-907c-4352-969d-f04b25dc7391-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd\" (UID: \"003334c6-907c-4352-969d-f04b25dc7391\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:05.077970 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:05.077597    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qtx\" (UniqueName: \"kubernetes.io/projected/003334c6-907c-4352-969d-f04b25dc7391-kube-api-access-n8qtx\") pod \"payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd\" (UID: \"003334c6-907c-4352-969d-f04b25dc7391\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:05.078292 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:05.078251    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003334c6-907c-4352-969d-f04b25dc7391-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd\" (UID: \"003334c6-907c-4352-969d-f04b25dc7391\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:05.085915 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:05.085889    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qtx\" (UniqueName: \"kubernetes.io/projected/003334c6-907c-4352-969d-f04b25dc7391-kube-api-access-n8qtx\") pod \"payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd\" (UID: \"003334c6-907c-4352-969d-f04b25dc7391\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:05.128788 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:05.128767    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:05.174890 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:05.174851    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:05.277743 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:05.277713    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"]
Jul 31 12:55:05.283628 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:55:05.283597    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ae1bc4_ed2a_45ba_abfd_2ad68e0c9cf4.slice/crio-74edd2a0e9bfd573451805aa853d7cacddab2bbc4a99ec74c42ae61d85923c30 WatchSource:0}: Error finding container 74edd2a0e9bfd573451805aa853d7cacddab2bbc4a99ec74c42ae61d85923c30: Status 404 returned error can't find the container with id 74edd2a0e9bfd573451805aa853d7cacddab2bbc4a99ec74c42ae61d85923c30
Jul 31 12:55:05.541322 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:05.541297    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"]
Jul 31 12:55:05.542974 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:55:05.542949    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003334c6_907c_4352_969d_f04b25dc7391.slice/crio-df5227501339921ba7d2abc524bf77f94c891f1ccdb8642825b062b9bf25d5a9 WatchSource:0}: Error finding container df5227501339921ba7d2abc524bf77f94c891f1ccdb8642825b062b9bf25d5a9: Status 404 returned error can't find the container with id df5227501339921ba7d2abc524bf77f94c891f1ccdb8642825b062b9bf25d5a9
Jul 31 12:55:06.060510 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.060471    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x" event={"ID":"e42917d5-6221-466b-a7a1-e3dc58168bf6","Type":"ContainerStarted","Data":"eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d"}
Jul 31 12:55:06.060736 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.060671    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:06.061926 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.061900    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd" event={"ID":"003334c6-907c-4352-969d-f04b25dc7391","Type":"ContainerStarted","Data":"a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260"}
Jul 31 12:55:06.062044 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.061947    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd" event={"ID":"003334c6-907c-4352-969d-f04b25dc7391","Type":"ContainerStarted","Data":"df5227501339921ba7d2abc524bf77f94c891f1ccdb8642825b062b9bf25d5a9"}
Jul 31 12:55:06.062044 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.061988    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:06.063397 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.063376    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9" event={"ID":"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4","Type":"ContainerStarted","Data":"cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab"}
Jul 31 12:55:06.063489 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.063416    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9" event={"ID":"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4","Type":"ContainerStarted","Data":"74edd2a0e9bfd573451805aa853d7cacddab2bbc4a99ec74c42ae61d85923c30"}
Jul 31 12:55:06.063489 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.063478    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:06.077211 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.077176    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x" podStartSLOduration=1.586578676 podStartE2EDuration="2.077162765s" totalImagesPullingTime="490.584089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:55:06.075219697 +0000 UTC m=+2394.062441237" watchObservedRunningTime="2026-07-31 12:55:06.077162765 +0000 UTC m=+2394.064384299"
Jul 31 12:55:06.090220 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.090183    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd" podStartSLOduration=2.090170228 podStartE2EDuration="2.090170228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:55:06.089065111 +0000 UTC m=+2394.076286651" watchObservedRunningTime="2026-07-31 12:55:06.090170228 +0000 UTC m=+2394.077391791"
Jul 31 12:55:06.104636 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:06.104594    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9" podStartSLOduration=2.104580092 podStartE2EDuration="2.104580092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:55:06.10379123 +0000 UTC m=+2394.091012771" watchObservedRunningTime="2026-07-31 12:55:06.104580092 +0000 UTC m=+2394.091801631"
Jul 31 12:55:12.074094 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:12.074068    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:55:17.070887 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:17.070854    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:55:17.071286 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:17.070934    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:55:17.700544 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:17.700204    2586 patch_prober.go:28] interesting pod/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": read tcp 10.132.0.2:58058->10.132.0.65:15021: read: connection reset by peer" start-of-body=
Jul 31 12:55:17.700544 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:17.700286    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": read tcp 10.132.0.2:58058->10.132.0.65:15021: read: connection reset by peer"
Jul 31 12:55:18.124085 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:18.123991    2586 generic.go:356] "Generic (PLEG): container finished" podID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerID="ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d" exitCode=0
Jul 31 12:55:18.124085 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:18.124067    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" event={"ID":"f914c43b-d069-46fb-88bf-5c8c3472d596","Type":"ContainerDied","Data":"ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d"}
Jul 31 12:55:18.124516 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:18.124460    2586 scope.go:117] "RemoveContainer" containerID="ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d"
Jul 31 12:55:18.125435 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:18.125405    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:55:18.125542 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:18.125489    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:55:18.125604 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:18.125538    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:55:19.131002 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:19.130972    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" event={"ID":"f914c43b-d069-46fb-88bf-5c8c3472d596","Type":"ContainerStarted","Data":"d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7"}
Jul 31 12:55:19.841096 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:19.840548    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:55:19.841096 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:19.840677    2586 patch_prober.go:28] interesting pod/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 12:55:19.841096 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:19.840721    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 12:55:20.839716 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:20.839676    2586 patch_prober.go:28] interesting pod/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 12:55:20.840231 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:20.839741    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 12:55:21.840008 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:21.839968    2586 patch_prober.go:28] interesting pod/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 12:55:21.840640 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:21.840606    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 12:55:22.840358 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:22.840311    2586 patch_prober.go:28] interesting pod/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 12:55:22.840874 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:22.840385    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 12:55:23.839658 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:23.839619    2586 patch_prober.go:28] interesting pod/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 12:55:23.839863 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:23.839690    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 12:55:24.843419 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:24.843389    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:55:24.843815 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:24.843638    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:55:24.844411 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:55:24.844391    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:56:12.523570 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:12.523536    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"]
Jul 31 12:56:12.524111 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:12.523774    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x" podUID="e42917d5-6221-466b-a7a1-e3dc58168bf6" containerName="maas-api" containerID="cri-o://eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d" gracePeriod=30
Jul 31 12:56:12.576596 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:12.576554    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"]
Jul 31 12:56:12.577007 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:12.576776    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9" podUID="86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4" containerName="payload-processing" containerID="cri-o://cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab" gracePeriod=30
Jul 31 12:56:12.582983 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:12.582947    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"]
Jul 31 12:56:12.583244 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:12.583185    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd" podUID="003334c6-907c-4352-969d-f04b25dc7391" containerName="payload-pre-processing" containerID="cri-o://a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260" gracePeriod=30
Jul 31 12:56:12.853453 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:12.853429    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:56:12.888220 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:12.888199    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:56:12.891457 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:12.891430    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:56:13.018582 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.018555    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sftpp\" (UniqueName: \"kubernetes.io/projected/e42917d5-6221-466b-a7a1-e3dc58168bf6-kube-api-access-sftpp\") pod \"e42917d5-6221-466b-a7a1-e3dc58168bf6\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") "
Jul 31 12:56:13.018759 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.018613    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk64x\" (UniqueName: \"kubernetes.io/projected/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-kube-api-access-rk64x\") pod \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\" (UID: \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\") "
Jul 31 12:56:13.018759 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.018640    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8qtx\" (UniqueName: \"kubernetes.io/projected/003334c6-907c-4352-969d-f04b25dc7391-kube-api-access-n8qtx\") pod \"003334c6-907c-4352-969d-f04b25dc7391\" (UID: \"003334c6-907c-4352-969d-f04b25dc7391\") "
Jul 31 12:56:13.018759 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.018666    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42917d5-6221-466b-a7a1-e3dc58168bf6-openshift-service-ca\") pod \"e42917d5-6221-466b-a7a1-e3dc58168bf6\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") "
Jul 31 12:56:13.018759 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.018745    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003334c6-907c-4352-969d-f04b25dc7391-plugins-config-volume\") pod \"003334c6-907c-4352-969d-f04b25dc7391\" (UID: \"003334c6-907c-4352-969d-f04b25dc7391\") "
Jul 31 12:56:13.019006 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.018777    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42917d5-6221-466b-a7a1-e3dc58168bf6-maas-api-tls\") pod \"e42917d5-6221-466b-a7a1-e3dc58168bf6\" (UID: \"e42917d5-6221-466b-a7a1-e3dc58168bf6\") "
Jul 31 12:56:13.019006 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.018802    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-plugins-config-volume\") pod \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\" (UID: \"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4\") "
Jul 31 12:56:13.019144 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.019115    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42917d5-6221-466b-a7a1-e3dc58168bf6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e42917d5-6221-466b-a7a1-e3dc58168bf6" (UID: "e42917d5-6221-466b-a7a1-e3dc58168bf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:56:13.019538 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.019510    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4" (UID: "86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:56:13.019648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.019596    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003334c6-907c-4352-969d-f04b25dc7391-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "003334c6-907c-4352-969d-f04b25dc7391" (UID: "003334c6-907c-4352-969d-f04b25dc7391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:56:13.021110 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.021072    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42917d5-6221-466b-a7a1-e3dc58168bf6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e42917d5-6221-466b-a7a1-e3dc58168bf6" (UID: "e42917d5-6221-466b-a7a1-e3dc58168bf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 12:56:13.021242 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.021214    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003334c6-907c-4352-969d-f04b25dc7391-kube-api-access-n8qtx" (OuterVolumeSpecName: "kube-api-access-n8qtx") pod "003334c6-907c-4352-969d-f04b25dc7391" (UID: "003334c6-907c-4352-969d-f04b25dc7391"). InnerVolumeSpecName "kube-api-access-n8qtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:56:13.021361 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.021251    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42917d5-6221-466b-a7a1-e3dc58168bf6-kube-api-access-sftpp" (OuterVolumeSpecName: "kube-api-access-sftpp") pod "e42917d5-6221-466b-a7a1-e3dc58168bf6" (UID: "e42917d5-6221-466b-a7a1-e3dc58168bf6"). InnerVolumeSpecName "kube-api-access-sftpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:56:13.021361 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.021302    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-kube-api-access-rk64x" (OuterVolumeSpecName: "kube-api-access-rk64x") pod "86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4" (UID: "86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4"). InnerVolumeSpecName "kube-api-access-rk64x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:56:13.119967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.119893    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003334c6-907c-4352-969d-f04b25dc7391-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:56:13.119967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.119919    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42917d5-6221-466b-a7a1-e3dc58168bf6-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:56:13.119967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.119929    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:56:13.119967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.119938    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sftpp\" (UniqueName: \"kubernetes.io/projected/e42917d5-6221-466b-a7a1-e3dc58168bf6-kube-api-access-sftpp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:56:13.119967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.119948    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk64x\" (UniqueName: \"kubernetes.io/projected/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4-kube-api-access-rk64x\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:56:13.119967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.119956    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8qtx\" (UniqueName: \"kubernetes.io/projected/003334c6-907c-4352-969d-f04b25dc7391-kube-api-access-n8qtx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:56:13.119967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.119965    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42917d5-6221-466b-a7a1-e3dc58168bf6-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:56:13.409918 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.409886    2586 generic.go:356] "Generic (PLEG): container finished" podID="e42917d5-6221-466b-a7a1-e3dc58168bf6" containerID="eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d" exitCode=0
Jul 31 12:56:13.410096 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.409932    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"
Jul 31 12:56:13.410096 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.409970    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x" event={"ID":"e42917d5-6221-466b-a7a1-e3dc58168bf6","Type":"ContainerDied","Data":"eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d"}
Jul 31 12:56:13.410096 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.410034    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x" event={"ID":"e42917d5-6221-466b-a7a1-e3dc58168bf6","Type":"ContainerDied","Data":"07143199cd51f6bf890288a32d16f0cb852dc885a1429e047a4aa74a4dabbad0"}
Jul 31 12:56:13.410096 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.410062    2586 scope.go:117] "RemoveContainer" containerID="eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d"
Jul 31 12:56:13.411301 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.411249    2586 generic.go:356] "Generic (PLEG): container finished" podID="003334c6-907c-4352-969d-f04b25dc7391" containerID="a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260" exitCode=0
Jul 31 12:56:13.411439 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.411306    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd" event={"ID":"003334c6-907c-4352-969d-f04b25dc7391","Type":"ContainerDied","Data":"a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260"}
Jul 31 12:56:13.411439 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.411357    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd" event={"ID":"003334c6-907c-4352-969d-f04b25dc7391","Type":"ContainerDied","Data":"df5227501339921ba7d2abc524bf77f94c891f1ccdb8642825b062b9bf25d5a9"}
Jul 31 12:56:13.411439 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.411310    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"
Jul 31 12:56:13.413218 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.412995    2586 generic.go:356] "Generic (PLEG): container finished" podID="86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4" containerID="cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab" exitCode=0
Jul 31 12:56:13.413218 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.413077    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9" event={"ID":"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4","Type":"ContainerDied","Data":"cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab"}
Jul 31 12:56:13.413218 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.413131    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"
Jul 31 12:56:13.413469 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.413227    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9" event={"ID":"86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4","Type":"ContainerDied","Data":"74edd2a0e9bfd573451805aa853d7cacddab2bbc4a99ec74c42ae61d85923c30"}
Jul 31 12:56:13.422139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.422116    2586 scope.go:117] "RemoveContainer" containerID="eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d"
Jul 31 12:56:13.422458 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:56:13.422435    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d\": container with ID starting with eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d not found: ID does not exist" containerID="eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d"
Jul 31 12:56:13.422572 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.422466    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d"} err="failed to get container status \"eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d\": rpc error: code = NotFound desc = could not find container \"eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d\": container with ID starting with eb34d73c9f02aee323f104ebcea3ef95ce160bd920414eed96a796a0d448c06d not found: ID does not exist"
Jul 31 12:56:13.422572 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.422487    2586 scope.go:117] "RemoveContainer" containerID="a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260"
Jul 31 12:56:13.430328 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.430313    2586 scope.go:117] "RemoveContainer" containerID="a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260"
Jul 31 12:56:13.430559 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:56:13.430540    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260\": container with ID starting with a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260 not found: ID does not exist" containerID="a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260"
Jul 31 12:56:13.430627 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.430565    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260"} err="failed to get container status \"a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260\": rpc error: code = NotFound desc = could not find container \"a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260\": container with ID starting with a4357211989afad3b9251e987a24e300786bb43eb0c34420946aa2e791354260 not found: ID does not exist"
Jul 31 12:56:13.430627 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.430579    2586 scope.go:117] "RemoveContainer" containerID="cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab"
Jul 31 12:56:13.438583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.438565    2586 scope.go:117] "RemoveContainer" containerID="cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab"
Jul 31 12:56:13.438849 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:56:13.438832    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab\": container with ID starting with cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab not found: ID does not exist" containerID="cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab"
Jul 31 12:56:13.438898 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.438856    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab"} err="failed to get container status \"cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab\": rpc error: code = NotFound desc = could not find container \"cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab\": container with ID starting with cd04cb49e45f9e51ebcab508374a2aac9fb57a6822c8b55a257028112f3498ab not found: ID does not exist"
Jul 31 12:56:13.439092 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.439077    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"]
Jul 31 12:56:13.442153 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.442135    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcf3ab8b-754cddc89b-j2kzd"]
Jul 31 12:56:13.454272 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.454242    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"]
Jul 31 12:56:13.457948 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.457928    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcf3ab8b-558745979-8kk9x"]
Jul 31 12:56:13.468622 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.468600    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"]
Jul 31 12:56:13.470697 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:13.470675    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bcf3ab8b-8674d9c6c8-4mzt9"]
Jul 31 12:56:14.483924 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:14.483891    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003334c6-907c-4352-969d-f04b25dc7391" path="/var/lib/kubelet/pods/003334c6-907c-4352-969d-f04b25dc7391/volumes"
Jul 31 12:56:14.484299 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:14.484232    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4" path="/var/lib/kubelet/pods/86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4/volumes"
Jul 31 12:56:14.484586 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:56:14.484573    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e42917d5-6221-466b-a7a1-e3dc58168bf6" path="/var/lib/kubelet/pods/e42917d5-6221-466b-a7a1-e3dc58168bf6/volumes"
Jul 31 12:57:03.629022 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:03.628991    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"]
Jul 31 12:57:03.629556 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:03.629227    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy" containerID="cri-o://d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7" gracePeriod=30
Jul 31 12:57:08.874920 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:08.874893    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:57:09.008858 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.008814    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-certs\") pod \"f914c43b-d069-46fb-88bf-5c8c3472d596\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") "
Jul 31 12:57:09.009064 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.008887    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-token\") pod \"f914c43b-d069-46fb-88bf-5c8c3472d596\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") "
Jul 31 12:57:09.009064 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.008907    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-envoy\") pod \"f914c43b-d069-46fb-88bf-5c8c3472d596\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") "
Jul 31 12:57:09.009064 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.008930    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-podinfo\") pod \"f914c43b-d069-46fb-88bf-5c8c3472d596\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") "
Jul 31 12:57:09.009064 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.008962    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-socket\") pod \"f914c43b-d069-46fb-88bf-5c8c3472d596\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") "
Jul 31 12:57:09.009064 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.008995    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlnk8\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-kube-api-access-qlnk8\") pod \"f914c43b-d069-46fb-88bf-5c8c3472d596\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") "
Jul 31 12:57:09.009064 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009029    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f914c43b-d069-46fb-88bf-5c8c3472d596-istiod-ca-cert\") pod \"f914c43b-d069-46fb-88bf-5c8c3472d596\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") "
Jul 31 12:57:09.009407 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009077    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-data\") pod \"f914c43b-d069-46fb-88bf-5c8c3472d596\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") "
Jul 31 12:57:09.009407 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009115    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-credential-socket\") pod \"f914c43b-d069-46fb-88bf-5c8c3472d596\" (UID: \"f914c43b-d069-46fb-88bf-5c8c3472d596\") "
Jul 31 12:57:09.009407 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009145    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f914c43b-d069-46fb-88bf-5c8c3472d596" (UID: "f914c43b-d069-46fb-88bf-5c8c3472d596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:57:09.009407 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009384    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-data" (OuterVolumeSpecName: "istio-data") pod "f914c43b-d069-46fb-88bf-5c8c3472d596" (UID: "f914c43b-d069-46fb-88bf-5c8c3472d596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:57:09.009628 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009460    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f914c43b-d069-46fb-88bf-5c8c3472d596-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f914c43b-d069-46fb-88bf-5c8c3472d596" (UID: "f914c43b-d069-46fb-88bf-5c8c3472d596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 12:57:09.009628 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009510    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:57:09.009628 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009584    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f914c43b-d069-46fb-88bf-5c8c3472d596" (UID: "f914c43b-d069-46fb-88bf-5c8c3472d596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:57:09.009628 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009604    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f914c43b-d069-46fb-88bf-5c8c3472d596" (UID: "f914c43b-d069-46fb-88bf-5c8c3472d596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:57:09.009628 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.009616    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:57:09.011278 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.011241    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f914c43b-d069-46fb-88bf-5c8c3472d596" (UID: "f914c43b-d069-46fb-88bf-5c8c3472d596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 12:57:09.011380 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.011283    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-token" (OuterVolumeSpecName: "istio-token") pod "f914c43b-d069-46fb-88bf-5c8c3472d596" (UID: "f914c43b-d069-46fb-88bf-5c8c3472d596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:57:09.011450 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.011434    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f914c43b-d069-46fb-88bf-5c8c3472d596" (UID: "f914c43b-d069-46fb-88bf-5c8c3472d596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 12:57:09.011577 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.011555    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-kube-api-access-qlnk8" (OuterVolumeSpecName: "kube-api-access-qlnk8") pod "f914c43b-d069-46fb-88bf-5c8c3472d596" (UID: "f914c43b-d069-46fb-88bf-5c8c3472d596"). InnerVolumeSpecName "kube-api-access-qlnk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 12:57:09.111127 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.111096    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f914c43b-d069-46fb-88bf-5c8c3472d596-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:57:09.111127 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.111121    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:57:09.111127 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.111133    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:57:09.111359 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.111142    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:57:09.111359 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.111150    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f914c43b-d069-46fb-88bf-5c8c3472d596-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:57:09.111359 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.111158    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f914c43b-d069-46fb-88bf-5c8c3472d596-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:57:09.111359 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.111167    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlnk8\" (UniqueName: \"kubernetes.io/projected/f914c43b-d069-46fb-88bf-5c8c3472d596-kube-api-access-qlnk8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 12:57:09.703335 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.703297    2586 generic.go:356] "Generic (PLEG): container finished" podID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerID="d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7" exitCode=0
Jul 31 12:57:09.703513 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.703345    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"
Jul 31 12:57:09.703513 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.703383    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" event={"ID":"f914c43b-d069-46fb-88bf-5c8c3472d596","Type":"ContainerDied","Data":"d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7"}
Jul 31 12:57:09.703513 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.703442    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857" event={"ID":"f914c43b-d069-46fb-88bf-5c8c3472d596","Type":"ContainerDied","Data":"5de9acc916cb73ce947dde7f840a35882f0db5f2e2481b99bed9946f3ecab6a0"}
Jul 31 12:57:09.703513 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.703466    2586 scope.go:117] "RemoveContainer" containerID="d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7"
Jul 31 12:57:09.713136 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.713120    2586 scope.go:117] "RemoveContainer" containerID="ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d"
Jul 31 12:57:09.720930 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.720905    2586 scope.go:117] "RemoveContainer" containerID="d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7"
Jul 31 12:57:09.721164 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:57:09.721140    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7\": container with ID starting with d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7 not found: ID does not exist" containerID="d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7"
Jul 31 12:57:09.721218 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.721174    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7"} err="failed to get container status \"d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7\": rpc error: code = NotFound desc = could not find container \"d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7\": container with ID starting with d4187eea8f40da36da881696c6827c0d762e258cefc3520dcf48681673796bd7 not found: ID does not exist"
Jul 31 12:57:09.721218 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.721192    2586 scope.go:117] "RemoveContainer" containerID="ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d"
Jul 31 12:57:09.721436 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:57:09.721400    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d\": container with ID starting with ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d not found: ID does not exist" containerID="ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d"
Jul 31 12:57:09.721483 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.721432    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d"} err="failed to get container status \"ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d\": rpc error: code = NotFound desc = could not find container \"ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d\": container with ID starting with ea3ab651cd1904c858edc2715644f8ba08c843d97182323cc8c110275d8ec87d not found: ID does not exist"
Jul 31 12:57:09.725210 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.725189    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"]
Jul 31 12:57:09.728967 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:09.728948    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bcf3ab8b-openshift-default-665bcd4888-s5857"]
Jul 31 12:57:10.484151 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:57:10.484119    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" path="/var/lib/kubelet/pods/f914c43b-d069-46fb-88bf-5c8c3472d596/volumes"
Jul 31 12:58:09.004187 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004157    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"]
Jul 31 12:58:09.004648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004562    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003334c6-907c-4352-969d-f04b25dc7391" containerName="payload-pre-processing"
Jul 31 12:58:09.004648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004576    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="003334c6-907c-4352-969d-f04b25dc7391" containerName="payload-pre-processing"
Jul 31 12:58:09.004648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004585    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy"
Jul 31 12:58:09.004648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004590    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy"
Jul 31 12:58:09.004648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004605    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e42917d5-6221-466b-a7a1-e3dc58168bf6" containerName="maas-api"
Jul 31 12:58:09.004648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004610    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e42917d5-6221-466b-a7a1-e3dc58168bf6" containerName="maas-api"
Jul 31 12:58:09.004648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004622    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4" containerName="payload-processing"
Jul 31 12:58:09.004648 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004627    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4" containerName="payload-processing"
Jul 31 12:58:09.004900 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004679    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy"
Jul 31 12:58:09.004900 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004686    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ae1bc4-ed2a-45ba-abfd-2ad68e0c9cf4" containerName="payload-processing"
Jul 31 12:58:09.004900 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004694    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="003334c6-907c-4352-969d-f04b25dc7391" containerName="payload-pre-processing"
Jul 31 12:58:09.004900 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.004700    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e42917d5-6221-466b-a7a1-e3dc58168bf6" containerName="maas-api"
Jul 31 12:58:09.007640 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.007623    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.010333 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.010309    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-20ab27-openshift-default-dockercfg-df5kr\""
Jul 31 12:58:09.029878 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.029851    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"]
Jul 31 12:58:09.107228 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.107193    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-data\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.107418 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.107247    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-token\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.107418 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.107344    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-credential-socket\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.107418 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.107373    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istiod-ca-cert\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.107418 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.107411    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-socket\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.107598 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.107434    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-certs\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.107598 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.107458    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-envoy\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.107598 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.107564    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-podinfo\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.107698 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.107603    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dr6\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-kube-api-access-k9dr6\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209001 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.208963    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-data\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209051    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-token\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209101    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-credential-socket\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209127    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istiod-ca-cert\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209176 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209176    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-socket\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209452 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209210    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-certs\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209452 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209234    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-envoy\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209452 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-podinfo\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209452 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209328    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dr6\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-kube-api-access-k9dr6\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209452 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-data\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209640 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209456    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-credential-socket\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209640 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209539    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-certs\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209826 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209802    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-socket\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.209937 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.209898    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istiod-ca-cert\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.211633 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.211600    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-envoy\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.211738 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.211719    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-podinfo\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.217581 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.217554    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-token\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.217986 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.217964    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dr6\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-kube-api-access-k9dr6\") pod \"e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") " pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.318218 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.318130    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:09.456373 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.456346    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"]
Jul 31 12:58:09.458189 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:58:09.458152    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcfb9df_3fab_48ae_b625_f2c8daf87258.slice/crio-0cb7ca690b6a1485d88825777ba8668ff474572a4ed9c7ff3c205540cddcb6e0 WatchSource:0}: Error finding container 0cb7ca690b6a1485d88825777ba8668ff474572a4ed9c7ff3c205540cddcb6e0: Status 404 returned error can't find the container with id 0cb7ca690b6a1485d88825777ba8668ff474572a4ed9c7ff3c205540cddcb6e0
Jul 31 12:58:09.460217 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.460172    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:58:09.460306 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.460286    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:58:09.460355 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.460337    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:58:09.990226 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.990188    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" event={"ID":"6bcfb9df-3fab-48ae-b625-f2c8daf87258","Type":"ContainerStarted","Data":"68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881"}
Jul 31 12:58:09.990450 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:09.990279    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" event={"ID":"6bcfb9df-3fab-48ae-b625-f2c8daf87258","Type":"ContainerStarted","Data":"0cb7ca690b6a1485d88825777ba8668ff474572a4ed9c7ff3c205540cddcb6e0"}
Jul 31 12:58:10.013181 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:10.013136    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" podStartSLOduration=2.013121088 podStartE2EDuration="2.013121088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:58:10.011075928 +0000 UTC m=+2577.998297483" watchObservedRunningTime="2026-07-31 12:58:10.013121088 +0000 UTC m=+2578.000342628"
Jul 31 12:58:10.319260 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:10.319181    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:10.324296 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:10.324245    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:10.995584 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:10.995554    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:10.996579 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:10.996562    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:12.996423 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:12.996387    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"]
Jul 31 12:58:12.996948 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:12.996931    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy"
Jul 31 12:58:12.997017 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:12.996949    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy"
Jul 31 12:58:12.997078 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:12.997044    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f914c43b-d069-46fb-88bf-5c8c3472d596" containerName="istio-proxy"
Jul 31 12:58:13.000478 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.000460    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.002827 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.002806    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-20ab27-serving-cert\""
Jul 31 12:58:13.008066 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.008044    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"]
Jul 31 12:58:13.044221 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.044194    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aacbbd8a-0a31-4086-b384-d6729388c064-maas-api-tls\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.044221 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.044223    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdxq\" (UniqueName: \"kubernetes.io/projected/aacbbd8a-0a31-4086-b384-d6729388c064-kube-api-access-mcdxq\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.044443 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.044375    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacbbd8a-0a31-4086-b384-d6729388c064-openshift-service-ca\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.145067 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.145037    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aacbbd8a-0a31-4086-b384-d6729388c064-maas-api-tls\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.145067 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.145069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdxq\" (UniqueName: \"kubernetes.io/projected/aacbbd8a-0a31-4086-b384-d6729388c064-kube-api-access-mcdxq\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.145283 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.145120    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacbbd8a-0a31-4086-b384-d6729388c064-openshift-service-ca\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.145283 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:58:13.145189    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-serving-cert: secret "maas-api-e2e-inf-a-20ab27-serving-cert" not found
Jul 31 12:58:13.145387 ip-10-0-136-35 kubenswrapper[2586]: E0731 12:58:13.145325    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aacbbd8a-0a31-4086-b384-d6729388c064-maas-api-tls podName:aacbbd8a-0a31-4086-b384-d6729388c064 nodeName:}" failed. No retries permitted until 2026-07-31 12:58:13.645302253 +0000 UTC m=+2581.632523791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aacbbd8a-0a31-4086-b384-d6729388c064-maas-api-tls") pod "maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv" (UID: "aacbbd8a-0a31-4086-b384-d6729388c064") : secret "maas-api-e2e-inf-a-20ab27-serving-cert" not found
Jul 31 12:58:13.145730 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.145711    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacbbd8a-0a31-4086-b384-d6729388c064-openshift-service-ca\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.155736 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.155709    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdxq\" (UniqueName: \"kubernetes.io/projected/aacbbd8a-0a31-4086-b384-d6729388c064-kube-api-access-mcdxq\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.650011 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.649979    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aacbbd8a-0a31-4086-b384-d6729388c064-maas-api-tls\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.652181 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.652164    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aacbbd8a-0a31-4086-b384-d6729388c064-maas-api-tls\") pod \"maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.703816 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.703792    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"]
Jul 31 12:58:13.711657 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.711630    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:13.714458 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.714440    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-20ab27-dockercfg-72hsk\""
Jul 31 12:58:13.714709 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.714691    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-20ab27\""
Jul 31 12:58:13.719360 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.719339    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"]
Jul 31 12:58:13.730461 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.730442    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"]
Jul 31 12:58:13.733692 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.733676    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:13.746202 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.746176    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"]
Jul 31 12:58:13.750985 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.750964    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5\" (UID: \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:13.751071 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.751004    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pz7m\" (UniqueName: \"kubernetes.io/projected/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-kube-api-access-7pz7m\") pod \"payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5\" (UID: \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:13.851947 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.851908    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc\" (UID: \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:13.851947 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.851949    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5\" (UID: \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:13.852202 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.852009    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j76s\" (UniqueName: \"kubernetes.io/projected/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-kube-api-access-7j76s\") pod \"payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc\" (UID: \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:13.852202 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.852062    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pz7m\" (UniqueName: \"kubernetes.io/projected/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-kube-api-access-7pz7m\") pod \"payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5\" (UID: \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:13.852656 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.852635    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5\" (UID: \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:13.861245 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.861220    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pz7m\" (UniqueName: \"kubernetes.io/projected/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-kube-api-access-7pz7m\") pod \"payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5\" (UID: \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:13.912960 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.912888    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:13.952603 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.952574    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc\" (UID: \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:13.952726 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.952620    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j76s\" (UniqueName: \"kubernetes.io/projected/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-kube-api-access-7j76s\") pod \"payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc\" (UID: \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:13.953428 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.953406    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc\" (UID: \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:13.961292 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:13.961184    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j76s\" (UniqueName: \"kubernetes.io/projected/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-kube-api-access-7j76s\") pod \"payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc\" (UID: \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:14.024335 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:14.024308    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:14.043164 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:14.042937    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:14.046260 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:14.046232    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"]
Jul 31 12:58:14.047944 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:58:14.047920    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacbbd8a_0a31_4086_b384_d6729388c064.slice/crio-49c7d079dfce1b4419fd2c6f160c03dcb434621649e8a53eb312c3753d558853 WatchSource:0}: Error finding container 49c7d079dfce1b4419fd2c6f160c03dcb434621649e8a53eb312c3753d558853: Status 404 returned error can't find the container with id 49c7d079dfce1b4419fd2c6f160c03dcb434621649e8a53eb312c3753d558853
Jul 31 12:58:14.178911 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:14.178814    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"]
Jul 31 12:58:14.181342 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:58:14.181252    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3c93fc_6a9c_4a2d_aac8_1c05f2f071c1.slice/crio-bc3857a0abaa09ea2f9cfc1a50051dc73c94778db65ba13fa99536d3044ba310 WatchSource:0}: Error finding container bc3857a0abaa09ea2f9cfc1a50051dc73c94778db65ba13fa99536d3044ba310: Status 404 returned error can't find the container with id bc3857a0abaa09ea2f9cfc1a50051dc73c94778db65ba13fa99536d3044ba310
Jul 31 12:58:14.200796 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:14.200770    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"]
Jul 31 12:58:14.203026 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:58:14.203003    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfad8bf1_031b_4e7c_b63e_2d8bf38a0a5c.slice/crio-8da19998f5feb56dfb62d031c52280669a5413ecebbed7e38e91b391521e6548 WatchSource:0}: Error finding container 8da19998f5feb56dfb62d031c52280669a5413ecebbed7e38e91b391521e6548: Status 404 returned error can't find the container with id 8da19998f5feb56dfb62d031c52280669a5413ecebbed7e38e91b391521e6548
Jul 31 12:58:15.017562 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.017527    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5" event={"ID":"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1","Type":"ContainerStarted","Data":"a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457"}
Jul 31 12:58:15.017799 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.017585    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5" event={"ID":"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1","Type":"ContainerStarted","Data":"bc3857a0abaa09ea2f9cfc1a50051dc73c94778db65ba13fa99536d3044ba310"}
Jul 31 12:58:15.017799 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.017713    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:15.019076 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.019054    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc" event={"ID":"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c","Type":"ContainerStarted","Data":"22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d"}
Jul 31 12:58:15.019205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.019097    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:15.019205 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.019118    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc" event={"ID":"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c","Type":"ContainerStarted","Data":"8da19998f5feb56dfb62d031c52280669a5413ecebbed7e38e91b391521e6548"}
Jul 31 12:58:15.020459 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.020438    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv" event={"ID":"aacbbd8a-0a31-4086-b384-d6729388c064","Type":"ContainerStarted","Data":"6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8"}
Jul 31 12:58:15.020583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.020479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv" event={"ID":"aacbbd8a-0a31-4086-b384-d6729388c064","Type":"ContainerStarted","Data":"49c7d079dfce1b4419fd2c6f160c03dcb434621649e8a53eb312c3753d558853"}
Jul 31 12:58:15.020583 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.020518    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:15.036204 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.036162    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5" podStartSLOduration=2.036149571 podStartE2EDuration="2.036149571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:58:15.034220984 +0000 UTC m=+2583.021442525" watchObservedRunningTime="2026-07-31 12:58:15.036149571 +0000 UTC m=+2583.023371111"
Jul 31 12:58:15.052284 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.052217    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv" podStartSLOduration=2.6471151969999998 podStartE2EDuration="3.052198143s" totalImagesPullingTime="405.082946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:58:15.051166275 +0000 UTC m=+2583.038387840" watchObservedRunningTime="2026-07-31 12:58:15.052198143 +0000 UTC m=+2583.039419685"
Jul 31 12:58:15.069626 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:15.069576    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc" podStartSLOduration=2.069558787 podStartE2EDuration="2.069558787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:58:15.0690333 +0000 UTC m=+2583.056254851" watchObservedRunningTime="2026-07-31 12:58:15.069558787 +0000 UTC m=+2583.056780331"
Jul 31 12:58:21.032386 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:21.032358    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 12:58:25.996793 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:25.996753    2586 patch_prober.go:28] interesting pod/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 31 12:58:25.997199 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:25.996808    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 31 12:58:26.028086 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:26.028061    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 12:58:26.028219 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:26.028147    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 12:58:26.089861 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:26.089835    2586 generic.go:356] "Generic (PLEG): container finished" podID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerID="68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881" exitCode=0
Jul 31 12:58:26.090016 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:26.089878    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" event={"ID":"6bcfb9df-3fab-48ae-b625-f2c8daf87258","Type":"ContainerDied","Data":"68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881"}
Jul 31 12:58:26.090242 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:26.090229    2586 scope.go:117] "RemoveContainer" containerID="68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881"
Jul 31 12:58:26.091373 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:26.091344    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:58:26.091481 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:26.091425    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:58:26.091543 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:26.091475    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 12:58:27.095666 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:27.095629    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" event={"ID":"6bcfb9df-3fab-48ae-b625-f2c8daf87258","Type":"ContainerStarted","Data":"60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448"}
Jul 31 12:58:27.319103 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:27.319068    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:27.319256 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:27.319229    2586 patch_prober.go:28] interesting pod/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 31 12:58:27.319344 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:27.319308    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 31 12:58:28.318947 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:28.318912    2586 patch_prober.go:28] interesting pod/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 31 12:58:28.319374 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:28.318966    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 31 12:58:29.319428 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:29.319393    2586 patch_prober.go:28] interesting pod/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 31 12:58:29.319830 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:29.319449    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 31 12:58:30.319370 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:30.319336    2586 patch_prober.go:28] interesting pod/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 31 12:58:30.319577 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:30.319396    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 31 12:58:31.319333 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:31.319298    2586 patch_prober.go:28] interesting pod/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 31 12:58:31.319513 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:31.319350    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 31 12:58:32.322874 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:32.322838    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:32.323357 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:32.323104    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:32.323906 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:32.323890    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 12:58:41.101683 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.101602    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"]
Jul 31 12:58:41.105237 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.105215    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.107508 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.107489    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e614c6-serving-cert\""
Jul 31 12:58:41.116298 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.116274    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"]
Jul 31 12:58:41.210552 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.210518    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4b2b\" (UniqueName: \"kubernetes.io/projected/37c7e503-8d45-49e7-9f82-c42bca758d71-kube-api-access-s4b2b\") pod \"maas-api-e2e-inf-b-e614c6-8588475886-p2hxf\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.210732 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.210575    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c7e503-8d45-49e7-9f82-c42bca758d71-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e614c6-8588475886-p2hxf\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.210732 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.210681    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37c7e503-8d45-49e7-9f82-c42bca758d71-maas-api-tls\") pod \"maas-api-e2e-inf-b-e614c6-8588475886-p2hxf\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.312202 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.312173    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4b2b\" (UniqueName: \"kubernetes.io/projected/37c7e503-8d45-49e7-9f82-c42bca758d71-kube-api-access-s4b2b\") pod \"maas-api-e2e-inf-b-e614c6-8588475886-p2hxf\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.312407 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.312213    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c7e503-8d45-49e7-9f82-c42bca758d71-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e614c6-8588475886-p2hxf\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.312407 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.312240    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37c7e503-8d45-49e7-9f82-c42bca758d71-maas-api-tls\") pod \"maas-api-e2e-inf-b-e614c6-8588475886-p2hxf\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.312844 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.312822    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c7e503-8d45-49e7-9f82-c42bca758d71-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e614c6-8588475886-p2hxf\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.314684 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.314662    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37c7e503-8d45-49e7-9f82-c42bca758d71-maas-api-tls\") pod \"maas-api-e2e-inf-b-e614c6-8588475886-p2hxf\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.320620 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.320602    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4b2b\" (UniqueName: \"kubernetes.io/projected/37c7e503-8d45-49e7-9f82-c42bca758d71-kube-api-access-s4b2b\") pod \"maas-api-e2e-inf-b-e614c6-8588475886-p2hxf\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.419431 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.419403    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:41.553361 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.553289    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"]
Jul 31 12:58:41.555561 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:58:41.555531    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c7e503_8d45_49e7_9f82_c42bca758d71.slice/crio-aa488e9ca0f0eb31c3231f297c1fcfe99f7efa2b7ef649360143add3e852315d WatchSource:0}: Error finding container aa488e9ca0f0eb31c3231f297c1fcfe99f7efa2b7ef649360143add3e852315d: Status 404 returned error can't find the container with id aa488e9ca0f0eb31c3231f297c1fcfe99f7efa2b7ef649360143add3e852315d
Jul 31 12:58:41.770722 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.770641    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"]
Jul 31 12:58:41.774077 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.774058    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 12:58:41.778046 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.778026    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e614c6\""
Jul 31 12:58:41.778139 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.778049    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e614c6-dockercfg-djks9\""
Jul 31 12:58:41.785808 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.785785    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"]
Jul 31 12:58:41.919811 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.919775    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2147535-5d49-48a5-8b4a-0870fc31fe41-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6\" (UID: \"d2147535-5d49-48a5-8b4a-0870fc31fe41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 12:58:41.919981 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:41.919846    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmv49\" (UniqueName: \"kubernetes.io/projected/d2147535-5d49-48a5-8b4a-0870fc31fe41-kube-api-access-lmv49\") pod \"payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6\" (UID: \"d2147535-5d49-48a5-8b4a-0870fc31fe41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 12:58:42.020610 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.020579    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2147535-5d49-48a5-8b4a-0870fc31fe41-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6\" (UID: \"d2147535-5d49-48a5-8b4a-0870fc31fe41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 12:58:42.020807 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.020762    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmv49\" (UniqueName: \"kubernetes.io/projected/d2147535-5d49-48a5-8b4a-0870fc31fe41-kube-api-access-lmv49\") pod \"payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6\" (UID: \"d2147535-5d49-48a5-8b4a-0870fc31fe41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 12:58:42.022294 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.021821    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2147535-5d49-48a5-8b4a-0870fc31fe41-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6\" (UID: \"d2147535-5d49-48a5-8b4a-0870fc31fe41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 12:58:42.029378 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.029337    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmv49\" (UniqueName: \"kubernetes.io/projected/d2147535-5d49-48a5-8b4a-0870fc31fe41-kube-api-access-lmv49\") pod \"payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6\" (UID: \"d2147535-5d49-48a5-8b4a-0870fc31fe41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 12:58:42.085109 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.085070    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 12:58:42.169100 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.169066    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf" event={"ID":"37c7e503-8d45-49e7-9f82-c42bca758d71","Type":"ContainerStarted","Data":"972db810481aad3e64dc967226352e9d94f0487986d64aa1748a04245e470bb6"}
Jul 31 12:58:42.169540 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.169155    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf" event={"ID":"37c7e503-8d45-49e7-9f82-c42bca758d71","Type":"ContainerStarted","Data":"aa488e9ca0f0eb31c3231f297c1fcfe99f7efa2b7ef649360143add3e852315d"}
Jul 31 12:58:42.169540 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.169198    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:42.193031 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.192981    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf" podStartSLOduration=0.828853457 podStartE2EDuration="1.192960357s" totalImagesPullingTime="364.1069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:58:42.188951068 +0000 UTC m=+2610.176172610" watchObservedRunningTime="2026-07-31 12:58:42.192960357 +0000 UTC m=+2610.180181897"
Jul 31 12:58:42.219235 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:42.219192    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"]
Jul 31 12:58:42.221379 ip-10-0-136-35 kubenswrapper[2586]: W0731 12:58:42.221352    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2147535_5d49_48a5_8b4a_0870fc31fe41.slice/crio-b882d9655b8a3276c48a2b4b68a6887e34ca70dd6a9584e1c5609d0b7abfddd3 WatchSource:0}: Error finding container b882d9655b8a3276c48a2b4b68a6887e34ca70dd6a9584e1c5609d0b7abfddd3: Status 404 returned error can't find the container with id b882d9655b8a3276c48a2b4b68a6887e34ca70dd6a9584e1c5609d0b7abfddd3
Jul 31 12:58:43.175413 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:43.175379    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6" event={"ID":"d2147535-5d49-48a5-8b4a-0870fc31fe41","Type":"ContainerStarted","Data":"f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360"}
Jul 31 12:58:43.175800 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:43.175434    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6" event={"ID":"d2147535-5d49-48a5-8b4a-0870fc31fe41","Type":"ContainerStarted","Data":"b882d9655b8a3276c48a2b4b68a6887e34ca70dd6a9584e1c5609d0b7abfddd3"}
Jul 31 12:58:43.192470 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:43.192422    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6" podStartSLOduration=2.1924072 podStartE2EDuration="2.1924072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 12:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 12:58:43.190878905 +0000 UTC m=+2611.178100444" watchObservedRunningTime="2026-07-31 12:58:43.1924072 +0000 UTC m=+2611.179628734"
Jul 31 12:58:44.181053 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:44.181026    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 12:58:48.181251 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:48.181217    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 12:58:55.188034 ip-10-0-136-35 kubenswrapper[2586]: I0731 12:58:55.188005    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 13:00:00.140731 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:00.140689    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j"]
Jul 31 13:00:00.144731 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:00.144713    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j"
Jul 31 13:00:00.147541 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:00.147513    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lxtlm\""
Jul 31 13:00:00.151987 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:00.151962    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j"]
Jul 31 13:00:00.163221 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:00.163196    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j"
Jul 31 13:00:00.310313 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:00.310287    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j"]
Jul 31 13:00:00.312776 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:00:00.312738    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0505ebe9_1d69_4334_91b7_b5cf2fc861db.slice/crio-627b24c3cdb88089ed6402dbf7596e8547bb294860e82f7675e35682840faead WatchSource:0}: Error finding container 627b24c3cdb88089ed6402dbf7596e8547bb294860e82f7675e35682840faead: Status 404 returned error can't find the container with id 627b24c3cdb88089ed6402dbf7596e8547bb294860e82f7675e35682840faead
Jul 31 13:00:00.558352 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:00.558323    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j" event={"ID":"0505ebe9-1d69-4334-91b7-b5cf2fc861db","Type":"ContainerStarted","Data":"627b24c3cdb88089ed6402dbf7596e8547bb294860e82f7675e35682840faead"}
Jul 31 13:00:01.564702 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:01.564668    2586 generic.go:356] "Generic (PLEG): container finished" podID="0505ebe9-1d69-4334-91b7-b5cf2fc861db" containerID="29d328a26a7e2206df4f97604b2c763857488e27202185640d4a6648c6587b70" exitCode=0
Jul 31 13:00:01.565088 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:01.564736    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j" event={"ID":"0505ebe9-1d69-4334-91b7-b5cf2fc861db","Type":"ContainerDied","Data":"29d328a26a7e2206df4f97604b2c763857488e27202185640d4a6648c6587b70"}
Jul 31 13:00:02.723947 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:02.723924    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j"
Jul 31 13:00:03.575967 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:03.575927    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j" event={"ID":"0505ebe9-1d69-4334-91b7-b5cf2fc861db","Type":"ContainerDied","Data":"627b24c3cdb88089ed6402dbf7596e8547bb294860e82f7675e35682840faead"}
Jul 31 13:00:03.575967 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:03.575963    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j"
Jul 31 13:00:03.575967 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:03.575980    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="627b24c3cdb88089ed6402dbf7596e8547bb294860e82f7675e35682840faead"
Jul 31 13:00:20.688300 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:20.688251    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"]
Jul 31 13:00:20.688722 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:20.688498    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv" podUID="aacbbd8a-0a31-4086-b384-d6729388c064" containerName="maas-api" containerID="cri-o://6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8" gracePeriod=30
Jul 31 13:00:20.738385 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:20.738352    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"]
Jul 31 13:00:20.738633 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:20.738607    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc" podUID="cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c" containerName="payload-processing" containerID="cri-o://22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d" gracePeriod=30
Jul 31 13:00:20.746162 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:20.746131    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"]
Jul 31 13:00:20.746415 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:20.746380    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5" podUID="1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1" containerName="payload-pre-processing" containerID="cri-o://a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457" gracePeriod=30
Jul 31 13:00:21.026659 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.026624    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv" podUID="aacbbd8a-0a31-4086-b384-d6729388c064" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.70:8443/health\": dial tcp 10.132.0.70:8443: connect: connection refused"
Jul 31 13:00:21.055620 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.055591    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 13:00:21.058352 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.058332    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 13:00:21.148947 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.148926    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 13:00:21.207126 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.207060    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j76s\" (UniqueName: \"kubernetes.io/projected/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-kube-api-access-7j76s\") pod \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\" (UID: \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\") "
Jul 31 13:00:21.207126 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.207100    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pz7m\" (UniqueName: \"kubernetes.io/projected/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-kube-api-access-7pz7m\") pod \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\" (UID: \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\") "
Jul 31 13:00:21.207347 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.207177    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-plugins-config-volume\") pod \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\" (UID: \"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c\") "
Jul 31 13:00:21.207347 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.207201    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-plugins-config-volume\") pod \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\" (UID: \"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1\") "
Jul 31 13:00:21.207545 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.207524    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c" (UID: "cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:00:21.207605 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.207572    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1" (UID: "1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:00:21.209090 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.209063    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-kube-api-access-7j76s" (OuterVolumeSpecName: "kube-api-access-7j76s") pod "cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c" (UID: "cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c"). InnerVolumeSpecName "kube-api-access-7j76s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:00:21.209196 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.209156    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-kube-api-access-7pz7m" (OuterVolumeSpecName: "kube-api-access-7pz7m") pod "1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1" (UID: "1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1"). InnerVolumeSpecName "kube-api-access-7pz7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:00:21.307949 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.307916    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcdxq\" (UniqueName: \"kubernetes.io/projected/aacbbd8a-0a31-4086-b384-d6729388c064-kube-api-access-mcdxq\") pod \"aacbbd8a-0a31-4086-b384-d6729388c064\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") "
Jul 31 13:00:21.308110 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.308031    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aacbbd8a-0a31-4086-b384-d6729388c064-maas-api-tls\") pod \"aacbbd8a-0a31-4086-b384-d6729388c064\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") "
Jul 31 13:00:21.308110 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.308057    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacbbd8a-0a31-4086-b384-d6729388c064-openshift-service-ca\") pod \"aacbbd8a-0a31-4086-b384-d6729388c064\" (UID: \"aacbbd8a-0a31-4086-b384-d6729388c064\") "
Jul 31 13:00:21.308391 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.308368    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:21.308391 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.308392    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j76s\" (UniqueName: \"kubernetes.io/projected/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-kube-api-access-7j76s\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:21.308530 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.308402    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pz7m\" (UniqueName: \"kubernetes.io/projected/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1-kube-api-access-7pz7m\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:21.308530 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.308413    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:21.308530 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.308475    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacbbd8a-0a31-4086-b384-d6729388c064-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aacbbd8a-0a31-4086-b384-d6729388c064" (UID: "aacbbd8a-0a31-4086-b384-d6729388c064"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:00:21.309959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.309942    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacbbd8a-0a31-4086-b384-d6729388c064-kube-api-access-mcdxq" (OuterVolumeSpecName: "kube-api-access-mcdxq") pod "aacbbd8a-0a31-4086-b384-d6729388c064" (UID: "aacbbd8a-0a31-4086-b384-d6729388c064"). InnerVolumeSpecName "kube-api-access-mcdxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:00:21.310027 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.310009    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacbbd8a-0a31-4086-b384-d6729388c064-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aacbbd8a-0a31-4086-b384-d6729388c064" (UID: "aacbbd8a-0a31-4086-b384-d6729388c064"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 13:00:21.409777 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.409735    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aacbbd8a-0a31-4086-b384-d6729388c064-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:21.409777 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.409771    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacbbd8a-0a31-4086-b384-d6729388c064-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:21.409777 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.409781    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdxq\" (UniqueName: \"kubernetes.io/projected/aacbbd8a-0a31-4086-b384-d6729388c064-kube-api-access-mcdxq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:21.669474 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.669442    2586 generic.go:356] "Generic (PLEG): container finished" podID="1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1" containerID="a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457" exitCode=0
Jul 31 13:00:21.669655 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.669487    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"
Jul 31 13:00:21.669655 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.669523    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5" event={"ID":"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1","Type":"ContainerDied","Data":"a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457"}
Jul 31 13:00:21.669655 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.669588    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5" event={"ID":"1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1","Type":"ContainerDied","Data":"bc3857a0abaa09ea2f9cfc1a50051dc73c94778db65ba13fa99536d3044ba310"}
Jul 31 13:00:21.669655 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.669622    2586 scope.go:117] "RemoveContainer" containerID="a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457"
Jul 31 13:00:21.670829 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.670806    2586 generic.go:356] "Generic (PLEG): container finished" podID="cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c" containerID="22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d" exitCode=0
Jul 31 13:00:21.670903 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.670864    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"
Jul 31 13:00:21.670903 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.670885    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc" event={"ID":"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c","Type":"ContainerDied","Data":"22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d"}
Jul 31 13:00:21.671004 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.670930    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc" event={"ID":"cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c","Type":"ContainerDied","Data":"8da19998f5feb56dfb62d031c52280669a5413ecebbed7e38e91b391521e6548"}
Jul 31 13:00:21.672404 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.672384    2586 generic.go:356] "Generic (PLEG): container finished" podID="aacbbd8a-0a31-4086-b384-d6729388c064" containerID="6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8" exitCode=0
Jul 31 13:00:21.672514 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.672457    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv" event={"ID":"aacbbd8a-0a31-4086-b384-d6729388c064","Type":"ContainerDied","Data":"6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8"}
Jul 31 13:00:21.672514 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.672477    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"
Jul 31 13:00:21.672514 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.672489    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv" event={"ID":"aacbbd8a-0a31-4086-b384-d6729388c064","Type":"ContainerDied","Data":"49c7d079dfce1b4419fd2c6f160c03dcb434621649e8a53eb312c3753d558853"}
Jul 31 13:00:21.682254 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.682238    2586 scope.go:117] "RemoveContainer" containerID="a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457"
Jul 31 13:00:21.682783 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:00:21.682754    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457\": container with ID starting with a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457 not found: ID does not exist" containerID="a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457"
Jul 31 13:00:21.682881 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.682792    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457"} err="failed to get container status \"a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457\": rpc error: code = NotFound desc = could not find container \"a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457\": container with ID starting with a359d2115db15150f2db30ea67472050884215a7c577b0d4c6fde2e285950457 not found: ID does not exist"
Jul 31 13:00:21.682881 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.682810    2586 scope.go:117] "RemoveContainer" containerID="22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d"
Jul 31 13:00:21.690759 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.690575    2586 scope.go:117] "RemoveContainer" containerID="22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d"
Jul 31 13:00:21.690988 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:00:21.690828    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d\": container with ID starting with 22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d not found: ID does not exist" containerID="22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d"
Jul 31 13:00:21.690988 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.690852    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d"} err="failed to get container status \"22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d\": rpc error: code = NotFound desc = could not find container \"22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d\": container with ID starting with 22db5ec1253b61ff74a632a4a9e8d35b55f06b61bec30859d5dd2dd5a4d54c2d not found: ID does not exist"
Jul 31 13:00:21.690988 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.690869    2586 scope.go:117] "RemoveContainer" containerID="6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8"
Jul 31 13:00:21.698213 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.698196    2586 scope.go:117] "RemoveContainer" containerID="6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8"
Jul 31 13:00:21.698494 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:00:21.698470    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8\": container with ID starting with 6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8 not found: ID does not exist" containerID="6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8"
Jul 31 13:00:21.698566 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.698504    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8"} err="failed to get container status \"6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8\": rpc error: code = NotFound desc = could not find container \"6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8\": container with ID starting with 6068209ac851d22d383f2e207b86d43cec29ea6ec7c516adc2d44230b7078cd8 not found: ID does not exist"
Jul 31 13:00:21.706666 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.706644    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"]
Jul 31 13:00:21.713362 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.713342    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ab27-bf5b8f6d4-9tzq5"]
Jul 31 13:00:21.733672 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.733654    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"]
Jul 31 13:00:21.737092 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.737072    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ab27-6d584db5b6-k5hfc"]
Jul 31 13:00:21.750132 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.750113    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"]
Jul 31 13:00:21.757847 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:21.757137    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ab27-5b9fcfbc5b-lzrlv"]
Jul 31 13:00:22.485021 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:22.484977    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1" path="/var/lib/kubelet/pods/1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1/volumes"
Jul 31 13:00:22.485496 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:22.485476    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aacbbd8a-0a31-4086-b384-d6729388c064" path="/var/lib/kubelet/pods/aacbbd8a-0a31-4086-b384-d6729388c064/volumes"
Jul 31 13:00:22.485857 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:22.485840    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c" path="/var/lib/kubelet/pods/cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c/volumes"
Jul 31 13:00:27.633149 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:27.633114    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"]
Jul 31 13:00:27.633560 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:27.633394    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy" containerID="cri-o://60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448" gracePeriod=30
Jul 31 13:00:32.326067 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.326028    2586 patch_prober.go:28] interesting pod/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 13:00:32.326573 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.326092    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 13:00:32.885222 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.885195    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 13:00:32.904435 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904405    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-certs\") pod \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") "
Jul 31 13:00:32.904435 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904436    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-token\") pod \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") "
Jul 31 13:00:32.904654 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904455    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-socket\") pod \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") "
Jul 31 13:00:32.904654 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904481    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-envoy\") pod \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") "
Jul 31 13:00:32.904654 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904501    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istiod-ca-cert\") pod \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") "
Jul 31 13:00:32.904654 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904537    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-data\") pod \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") "
Jul 31 13:00:32.904654 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904552    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-podinfo\") pod \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") "
Jul 31 13:00:32.904654 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904601    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-credential-socket\") pod \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") "
Jul 31 13:00:32.904654 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904639    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9dr6\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-kube-api-access-k9dr6\") pod \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\" (UID: \"6bcfb9df-3fab-48ae-b625-f2c8daf87258\") "
Jul 31 13:00:32.905025 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904727    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6bcfb9df-3fab-48ae-b625-f2c8daf87258" (UID: "6bcfb9df-3fab-48ae-b625-f2c8daf87258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:00:32.905025 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904917    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-data" (OuterVolumeSpecName: "istio-data") pod "6bcfb9df-3fab-48ae-b625-f2c8daf87258" (UID: "6bcfb9df-3fab-48ae-b625-f2c8daf87258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:00:32.905025 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.904940    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6bcfb9df-3fab-48ae-b625-f2c8daf87258" (UID: "6bcfb9df-3fab-48ae-b625-f2c8daf87258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:00:32.905193 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.905028    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:32.905193 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.905045    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:32.905193 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.905082    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:32.905193 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.905107    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6bcfb9df-3fab-48ae-b625-f2c8daf87258" (UID: "6bcfb9df-3fab-48ae-b625-f2c8daf87258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:00:32.905402 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.905209    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6bcfb9df-3fab-48ae-b625-f2c8daf87258" (UID: "6bcfb9df-3fab-48ae-b625-f2c8daf87258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:00:32.906750 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.906725    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-kube-api-access-k9dr6" (OuterVolumeSpecName: "kube-api-access-k9dr6") pod "6bcfb9df-3fab-48ae-b625-f2c8daf87258" (UID: "6bcfb9df-3fab-48ae-b625-f2c8daf87258"). InnerVolumeSpecName "kube-api-access-k9dr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:00:32.906860 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.906844    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-token" (OuterVolumeSpecName: "istio-token") pod "6bcfb9df-3fab-48ae-b625-f2c8daf87258" (UID: "6bcfb9df-3fab-48ae-b625-f2c8daf87258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:00:32.907008 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.906990    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6bcfb9df-3fab-48ae-b625-f2c8daf87258" (UID: "6bcfb9df-3fab-48ae-b625-f2c8daf87258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:00:32.907107 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:32.907007    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6bcfb9df-3fab-48ae-b625-f2c8daf87258" (UID: "6bcfb9df-3fab-48ae-b625-f2c8daf87258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 13:00:33.005878 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.005835    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:33.005878 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.005867    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:33.005878 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.005883    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dr6\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-kube-api-access-k9dr6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:33.006122 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.005896    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:33.006122 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.005908    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:33.006122 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.005918    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcfb9df-3fab-48ae-b625-f2c8daf87258-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:33.732751 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.732715    2586 generic.go:356] "Generic (PLEG): container finished" podID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerID="60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448" exitCode=0
Jul 31 13:00:33.733225 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.732775    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"
Jul 31 13:00:33.733225 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.732795    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" event={"ID":"6bcfb9df-3fab-48ae-b625-f2c8daf87258","Type":"ContainerDied","Data":"60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448"}
Jul 31 13:00:33.733225 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.732850    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg" event={"ID":"6bcfb9df-3fab-48ae-b625-f2c8daf87258","Type":"ContainerDied","Data":"0cb7ca690b6a1485d88825777ba8668ff474572a4ed9c7ff3c205540cddcb6e0"}
Jul 31 13:00:33.733225 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.732871    2586 scope.go:117] "RemoveContainer" containerID="60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448"
Jul 31 13:00:33.741719 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.741702    2586 scope.go:117] "RemoveContainer" containerID="68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881"
Jul 31 13:00:33.749543 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.749527    2586 scope.go:117] "RemoveContainer" containerID="60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448"
Jul 31 13:00:33.749780 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:00:33.749762    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448\": container with ID starting with 60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448 not found: ID does not exist" containerID="60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448"
Jul 31 13:00:33.749841 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.749788    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448"} err="failed to get container status \"60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448\": rpc error: code = NotFound desc = could not find container \"60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448\": container with ID starting with 60a8f9268d06c9183fcd133aff12731d697991498a8ab7e03d4ca824617fd448 not found: ID does not exist"
Jul 31 13:00:33.749841 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.749802    2586 scope.go:117] "RemoveContainer" containerID="68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881"
Jul 31 13:00:33.750057 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:00:33.750039    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881\": container with ID starting with 68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881 not found: ID does not exist" containerID="68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881"
Jul 31 13:00:33.750104 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.750062    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881"} err="failed to get container status \"68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881\": rpc error: code = NotFound desc = could not find container \"68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881\": container with ID starting with 68e2a87a5ac21ff6a1168927b9821888924ac3fe5877d2d540f0d19b2cd4c881 not found: ID does not exist"
Jul 31 13:00:33.759472 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.759453    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"]
Jul 31 13:00:33.763533 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:33.763511    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-20ab27-openshift-default-6587d4f98b-27tdg"]
Jul 31 13:00:34.484856 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:34.484820    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" path="/var/lib/kubelet/pods/6bcfb9df-3fab-48ae-b625-f2c8daf87258/volumes"
Jul 31 13:00:38.603198 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.603168    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"]
Jul 31 13:00:38.603630 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.603385    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf" podUID="37c7e503-8d45-49e7-9f82-c42bca758d71" containerName="maas-api" containerID="cri-o://972db810481aad3e64dc967226352e9d94f0487986d64aa1748a04245e470bb6" gracePeriod=30
Jul 31 13:00:38.639297 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.639257    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j"]
Jul 31 13:00:38.644177 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.644158    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e614c6-29758380-t6n5j"]
Jul 31 13:00:38.681446 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.681420    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"]
Jul 31 13:00:38.681677 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.681648    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6" podUID="d2147535-5d49-48a5-8b4a-0870fc31fe41" containerName="payload-pre-processing" containerID="cri-o://f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360" gracePeriod=30
Jul 31 13:00:38.758876 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.758845    2586 generic.go:356] "Generic (PLEG): container finished" podID="37c7e503-8d45-49e7-9f82-c42bca758d71" containerID="972db810481aad3e64dc967226352e9d94f0487986d64aa1748a04245e470bb6" exitCode=0
Jul 31 13:00:38.759005 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.758927    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf" event={"ID":"37c7e503-8d45-49e7-9f82-c42bca758d71","Type":"ContainerDied","Data":"972db810481aad3e64dc967226352e9d94f0487986d64aa1748a04245e470bb6"}
Jul 31 13:00:38.932586 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.932565    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 13:00:38.958730 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.958698    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37c7e503-8d45-49e7-9f82-c42bca758d71-maas-api-tls\") pod \"37c7e503-8d45-49e7-9f82-c42bca758d71\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") "
Jul 31 13:00:38.958862 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.958758    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4b2b\" (UniqueName: \"kubernetes.io/projected/37c7e503-8d45-49e7-9f82-c42bca758d71-kube-api-access-s4b2b\") pod \"37c7e503-8d45-49e7-9f82-c42bca758d71\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") "
Jul 31 13:00:38.958862 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.958825    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c7e503-8d45-49e7-9f82-c42bca758d71-openshift-service-ca\") pod \"37c7e503-8d45-49e7-9f82-c42bca758d71\" (UID: \"37c7e503-8d45-49e7-9f82-c42bca758d71\") "
Jul 31 13:00:38.959456 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.959355    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c7e503-8d45-49e7-9f82-c42bca758d71-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37c7e503-8d45-49e7-9f82-c42bca758d71" (UID: "37c7e503-8d45-49e7-9f82-c42bca758d71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:00:38.961013 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.960989    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c7e503-8d45-49e7-9f82-c42bca758d71-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37c7e503-8d45-49e7-9f82-c42bca758d71" (UID: "37c7e503-8d45-49e7-9f82-c42bca758d71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 13:00:38.968058 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.961224    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c7e503-8d45-49e7-9f82-c42bca758d71-kube-api-access-s4b2b" (OuterVolumeSpecName: "kube-api-access-s4b2b") pod "37c7e503-8d45-49e7-9f82-c42bca758d71" (UID: "37c7e503-8d45-49e7-9f82-c42bca758d71"). InnerVolumeSpecName "kube-api-access-s4b2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:00:38.968058 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:38.962963    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 13:00:39.060227 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.060193    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2147535-5d49-48a5-8b4a-0870fc31fe41-plugins-config-volume\") pod \"d2147535-5d49-48a5-8b4a-0870fc31fe41\" (UID: \"d2147535-5d49-48a5-8b4a-0870fc31fe41\") "
Jul 31 13:00:39.060424 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.060312    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmv49\" (UniqueName: \"kubernetes.io/projected/d2147535-5d49-48a5-8b4a-0870fc31fe41-kube-api-access-lmv49\") pod \"d2147535-5d49-48a5-8b4a-0870fc31fe41\" (UID: \"d2147535-5d49-48a5-8b4a-0870fc31fe41\") "
Jul 31 13:00:39.060479 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.060468    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37c7e503-8d45-49e7-9f82-c42bca758d71-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:39.060517 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.060483    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4b2b\" (UniqueName: \"kubernetes.io/projected/37c7e503-8d45-49e7-9f82-c42bca758d71-kube-api-access-s4b2b\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:39.060517 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.060494    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c7e503-8d45-49e7-9f82-c42bca758d71-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:39.060669 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.060631    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2147535-5d49-48a5-8b4a-0870fc31fe41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2147535-5d49-48a5-8b4a-0870fc31fe41" (UID: "d2147535-5d49-48a5-8b4a-0870fc31fe41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:00:39.062289 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.062250    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2147535-5d49-48a5-8b4a-0870fc31fe41-kube-api-access-lmv49" (OuterVolumeSpecName: "kube-api-access-lmv49") pod "d2147535-5d49-48a5-8b4a-0870fc31fe41" (UID: "d2147535-5d49-48a5-8b4a-0870fc31fe41"). InnerVolumeSpecName "kube-api-access-lmv49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:00:39.161392 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.161362    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmv49\" (UniqueName: \"kubernetes.io/projected/d2147535-5d49-48a5-8b4a-0870fc31fe41-kube-api-access-lmv49\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:39.161392 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.161387    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2147535-5d49-48a5-8b4a-0870fc31fe41-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:00:39.765291 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.765233    2586 generic.go:356] "Generic (PLEG): container finished" podID="d2147535-5d49-48a5-8b4a-0870fc31fe41" containerID="f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360" exitCode=0
Jul 31 13:00:39.765728 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.765307    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"
Jul 31 13:00:39.765728 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.765324    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6" event={"ID":"d2147535-5d49-48a5-8b4a-0870fc31fe41","Type":"ContainerDied","Data":"f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360"}
Jul 31 13:00:39.765728 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.765390    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6" event={"ID":"d2147535-5d49-48a5-8b4a-0870fc31fe41","Type":"ContainerDied","Data":"b882d9655b8a3276c48a2b4b68a6887e34ca70dd6a9584e1c5609d0b7abfddd3"}
Jul 31 13:00:39.765728 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.765419    2586 scope.go:117] "RemoveContainer" containerID="f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360"
Jul 31 13:00:39.766853 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.766833    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"
Jul 31 13:00:39.766980 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.766837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf" event={"ID":"37c7e503-8d45-49e7-9f82-c42bca758d71","Type":"ContainerDied","Data":"aa488e9ca0f0eb31c3231f297c1fcfe99f7efa2b7ef649360143add3e852315d"}
Jul 31 13:00:39.775356 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.775341    2586 scope.go:117] "RemoveContainer" containerID="f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360"
Jul 31 13:00:39.775613 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:00:39.775597    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360\": container with ID starting with f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360 not found: ID does not exist" containerID="f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360"
Jul 31 13:00:39.775702 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.775620    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360"} err="failed to get container status \"f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360\": rpc error: code = NotFound desc = could not find container \"f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360\": container with ID starting with f46c8768318651985f87e64a7d090d262a9bc4af69afb5e07a756a2a85363360 not found: ID does not exist"
Jul 31 13:00:39.775702 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.775646    2586 scope.go:117] "RemoveContainer" containerID="972db810481aad3e64dc967226352e9d94f0487986d64aa1748a04245e470bb6"
Jul 31 13:00:39.788416 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.788397    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"]
Jul 31 13:00:39.792562 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.792536    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e614c6-8596d55b7b-frfw6"]
Jul 31 13:00:39.803245 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.803221    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"]
Jul 31 13:00:39.809079 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:39.809061    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e614c6-8588475886-p2hxf"]
Jul 31 13:00:40.484546 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:40.484513    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0505ebe9-1d69-4334-91b7-b5cf2fc861db" path="/var/lib/kubelet/pods/0505ebe9-1d69-4334-91b7-b5cf2fc861db/volumes"
Jul 31 13:00:40.484959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:40.484939    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37c7e503-8d45-49e7-9f82-c42bca758d71" path="/var/lib/kubelet/pods/37c7e503-8d45-49e7-9f82-c42bca758d71/volumes"
Jul 31 13:00:40.485352 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:40.485334    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2147535-5d49-48a5-8b4a-0870fc31fe41" path="/var/lib/kubelet/pods/d2147535-5d49-48a5-8b4a-0870fc31fe41/volumes"
Jul 31 13:00:46.435571 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.435540    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"]
Jul 31 13:00:46.436092 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436075    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy"
Jul 31 13:00:46.436172 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436094    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy"
Jul 31 13:00:46.436172 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436109    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0505ebe9-1d69-4334-91b7-b5cf2fc861db" containerName="cleanup"
Jul 31 13:00:46.436172 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436117    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0505ebe9-1d69-4334-91b7-b5cf2fc861db" containerName="cleanup"
Jul 31 13:00:46.436172 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436135    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37c7e503-8d45-49e7-9f82-c42bca758d71" containerName="maas-api"
Jul 31 13:00:46.436172 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436144    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="37c7e503-8d45-49e7-9f82-c42bca758d71" containerName="maas-api"
Jul 31 13:00:46.436172 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436154    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2147535-5d49-48a5-8b4a-0870fc31fe41" containerName="payload-pre-processing"
Jul 31 13:00:46.436172 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436163    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2147535-5d49-48a5-8b4a-0870fc31fe41" containerName="payload-pre-processing"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436176    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c" containerName="payload-processing"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436184    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c" containerName="payload-processing"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436204    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1" containerName="payload-pre-processing"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436213    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1" containerName="payload-pre-processing"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436223    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aacbbd8a-0a31-4086-b384-d6729388c064" containerName="maas-api"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436231    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aacbbd8a-0a31-4086-b384-d6729388c064" containerName="maas-api"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436357    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436371    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0505ebe9-1d69-4334-91b7-b5cf2fc861db" containerName="cleanup"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436382    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a3c93fc-6a9c-4a2d-aac8-1c05f2f071c1" containerName="payload-pre-processing"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436390    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2147535-5d49-48a5-8b4a-0870fc31fe41" containerName="payload-pre-processing"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436398    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="37c7e503-8d45-49e7-9f82-c42bca758d71" containerName="maas-api"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436408    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfad8bf1-031b-4e7c-b63e-2d8bf38a0a5c" containerName="payload-processing"
Jul 31 13:00:46.436535 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.436418    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aacbbd8a-0a31-4086-b384-d6729388c064" containerName="maas-api"
Jul 31 13:00:46.441124 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.441106    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.443910 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.443888    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b0c802-openshift-default-dockercfg-f5qjv\""
Jul 31 13:00:46.453171 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.453150    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"]
Jul 31 13:00:46.521056 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.521017    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e26fe4-e454-4c99-8bec-90e0852042b3-istiod-ca-cert\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.521056 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.521060    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-certs\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.521312 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.521099    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-envoy\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.521312 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.521159    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-socket\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.521312 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.521192    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-credential-socket\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.521312 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.521215    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-data\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.521312 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.521250    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-token\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.521534 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.521316    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqrlx\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-kube-api-access-sqrlx\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.521534 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.521367    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-podinfo\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622102 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-envoy\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622343 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622132    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-socket\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622343 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622161    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-credential-socket\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622343 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622185    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-data\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622343 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622207    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-token\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622343 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622230    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqrlx\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-kube-api-access-sqrlx\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622635 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622426    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-podinfo\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622635 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622515    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e26fe4-e454-4c99-8bec-90e0852042b3-istiod-ca-cert\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622635 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622553    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-certs\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622635 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622618    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-data\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622832 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622687    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-credential-socket\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622832 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622748    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-socket\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.622940 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.622886    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-certs\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.623158 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.623139    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e26fe4-e454-4c99-8bec-90e0852042b3-istiod-ca-cert\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.624602 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.624582    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-envoy\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.624813 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.624791    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-podinfo\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.630226 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.630199    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-token\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.630442 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.630425    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqrlx\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-kube-api-access-sqrlx\") pod \"e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") " pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.752402 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.752322    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:46.910658 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.910635    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"]
Jul 31 13:00:46.912591 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:00:46.912552    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e26fe4_e454_4c99_8bec_90e0852042b3.slice/crio-b74e00424d21b7521a82db10f372cb02eaa755df4fbfa7d4dd962052b39fbd36 WatchSource:0}: Error finding container b74e00424d21b7521a82db10f372cb02eaa755df4fbfa7d4dd962052b39fbd36: Status 404 returned error can't find the container with id b74e00424d21b7521a82db10f372cb02eaa755df4fbfa7d4dd962052b39fbd36
Jul 31 13:00:46.914876 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.914848    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:00:46.914949 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.914913    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:00:46.914949 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:46.914940    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:00:47.819491 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:47.819458    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" event={"ID":"e9e26fe4-e454-4c99-8bec-90e0852042b3","Type":"ContainerStarted","Data":"4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22"}
Jul 31 13:00:47.819924 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:47.819515    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" event={"ID":"e9e26fe4-e454-4c99-8bec-90e0852042b3","Type":"ContainerStarted","Data":"b74e00424d21b7521a82db10f372cb02eaa755df4fbfa7d4dd962052b39fbd36"}
Jul 31 13:00:47.844222 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:47.844170    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" podStartSLOduration=1.844151583 podStartE2EDuration="1.844151583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:00:47.84368268 +0000 UTC m=+2735.830904219" watchObservedRunningTime="2026-07-31 13:00:47.844151583 +0000 UTC m=+2735.831373124"
Jul 31 13:00:48.753481 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:48.753450    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:48.758763 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:48.758737    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:48.824823 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:48.824792    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:48.825868 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:48.825853    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:00:50.993428 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:50.993393    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"]
Jul 31 13:00:50.993946 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:50.993931    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy"
Jul 31 13:00:50.994025 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:50.993948    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy"
Jul 31 13:00:50.994113 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:50.994037    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcfb9df-3fab-48ae-b625-f2c8daf87258" containerName="istio-proxy"
Jul 31 13:00:50.997167 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:50.997145    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:50.999447 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:50.999425    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b0c802-serving-cert\""
Jul 31 13:00:51.008024 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.007993    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"]
Jul 31 13:00:51.062075 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.062041    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.062227 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.062095    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmv4c\" (UniqueName: \"kubernetes.io/projected/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-kube-api-access-gmv4c\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.062227 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.062172    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-maas-api-tls\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.163513 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.163469    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-maas-api-tls\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.163676 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.163570    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.163676 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:00:51.163620    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-serving-cert: secret "maas-api-e2e-api-a-b0c802-serving-cert" not found
Jul 31 13:00:51.163676 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.163633    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmv4c\" (UniqueName: \"kubernetes.io/projected/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-kube-api-access-gmv4c\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.163819 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:00:51.163685    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-maas-api-tls podName:a0352937-a4e7-4ee7-8f57-1078a8ae5d9e nodeName:}" failed. No retries permitted until 2026-07-31 13:00:51.663666346 +0000 UTC m=+2739.650887878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-maas-api-tls") pod "maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c" (UID: "a0352937-a4e7-4ee7-8f57-1078a8ae5d9e") : secret "maas-api-e2e-api-a-b0c802-serving-cert" not found
Jul 31 13:00:51.164279 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.164244    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.174246 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.174222    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmv4c\" (UniqueName: \"kubernetes.io/projected/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-kube-api-access-gmv4c\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.555474 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.555437    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"]
Jul 31 13:00:51.559090 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.559067    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:00:51.561389 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.561365    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b0c802\""
Jul 31 13:00:51.570082 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.570058    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"]
Jul 31 13:00:51.589513 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.589490    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"]
Jul 31 13:00:51.592735 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.592720    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:00:51.595175 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.595157    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b0c802-dockercfg-hfrwh\""
Jul 31 13:00:51.603524 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.603502    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"]
Jul 31 13:00:51.668214 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.668186    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtv75\" (UniqueName: \"kubernetes.io/projected/d0e46e26-ffc8-4aef-958d-6ce4df70f878-kube-api-access-qtv75\") pod \"payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2\" (UID: \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:00:51.668396 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.668276    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e46e26-ffc8-4aef-958d-6ce4df70f878-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2\" (UID: \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:00:51.668396 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.668361    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e213fb-c3ba-4a80-97ca-9022fea600ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm\" (UID: \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:00:51.668396 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.668380    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8vb\" (UniqueName: \"kubernetes.io/projected/f0e213fb-c3ba-4a80-97ca-9022fea600ab-kube-api-access-sj8vb\") pod \"payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm\" (UID: \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:00:51.668510 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.668420    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-maas-api-tls\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.670695 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.670671    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-maas-api-tls\") pod \"maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:51.769173 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.769140    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtv75\" (UniqueName: \"kubernetes.io/projected/d0e46e26-ffc8-4aef-958d-6ce4df70f878-kube-api-access-qtv75\") pod \"payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2\" (UID: \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:00:51.769365 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.769220    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e46e26-ffc8-4aef-958d-6ce4df70f878-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2\" (UID: \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:00:51.769365 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.769247    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e213fb-c3ba-4a80-97ca-9022fea600ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm\" (UID: \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:00:51.769365 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.769288    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8vb\" (UniqueName: \"kubernetes.io/projected/f0e213fb-c3ba-4a80-97ca-9022fea600ab-kube-api-access-sj8vb\") pod \"payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm\" (UID: \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:00:51.770016 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.769992    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e213fb-c3ba-4a80-97ca-9022fea600ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm\" (UID: \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:00:51.770107 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.769986    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e46e26-ffc8-4aef-958d-6ce4df70f878-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2\" (UID: \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:00:51.780813 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.780782    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8vb\" (UniqueName: \"kubernetes.io/projected/f0e213fb-c3ba-4a80-97ca-9022fea600ab-kube-api-access-sj8vb\") pod \"payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm\" (UID: \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:00:51.781327 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.781309    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtv75\" (UniqueName: \"kubernetes.io/projected/d0e46e26-ffc8-4aef-958d-6ce4df70f878-kube-api-access-qtv75\") pod \"payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2\" (UID: \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:00:51.870388 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.870306    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:00:51.903357 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.903329    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:00:51.910383 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:51.910356    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:52.038075 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.038048    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"]
Jul 31 13:00:52.038849 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:00:52.038825    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e213fb_c3ba_4a80_97ca_9022fea600ab.slice/crio-1055fd3d934e18968207f78b2b8b160f7aea0e42bc5e1c826c6c16921593f532 WatchSource:0}: Error finding container 1055fd3d934e18968207f78b2b8b160f7aea0e42bc5e1c826c6c16921593f532: Status 404 returned error can't find the container with id 1055fd3d934e18968207f78b2b8b160f7aea0e42bc5e1c826c6c16921593f532
Jul 31 13:00:52.095510 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.095481    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"]
Jul 31 13:00:52.097325 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:00:52.097298    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0352937_a4e7_4ee7_8f57_1078a8ae5d9e.slice/crio-ca360bcf5a15846a128521bd5467e6547b874e3a173ea797ef4b4fdc287aa51e WatchSource:0}: Error finding container ca360bcf5a15846a128521bd5467e6547b874e3a173ea797ef4b4fdc287aa51e: Status 404 returned error can't find the container with id ca360bcf5a15846a128521bd5467e6547b874e3a173ea797ef4b4fdc287aa51e
Jul 31 13:00:52.098761 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.098741    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 13:00:52.273016 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.272994    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"]
Jul 31 13:00:52.275772 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:00:52.275744    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e46e26_ffc8_4aef_958d_6ce4df70f878.slice/crio-95b98b8e37bf2a9c3c885306d982608d98a7d2ff767e5dd183260b5fe9652063 WatchSource:0}: Error finding container 95b98b8e37bf2a9c3c885306d982608d98a7d2ff767e5dd183260b5fe9652063: Status 404 returned error can't find the container with id 95b98b8e37bf2a9c3c885306d982608d98a7d2ff767e5dd183260b5fe9652063
Jul 31 13:00:52.847326 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.847224    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c" event={"ID":"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e","Type":"ContainerStarted","Data":"99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908"}
Jul 31 13:00:52.847326 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.847302    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:00:52.847326 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.847324    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c" event={"ID":"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e","Type":"ContainerStarted","Data":"ca360bcf5a15846a128521bd5467e6547b874e3a173ea797ef4b4fdc287aa51e"}
Jul 31 13:00:52.848996 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.848962    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2" event={"ID":"d0e46e26-ffc8-4aef-958d-6ce4df70f878","Type":"ContainerStarted","Data":"5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af"}
Jul 31 13:00:52.849131 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.849011    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2" event={"ID":"d0e46e26-ffc8-4aef-958d-6ce4df70f878","Type":"ContainerStarted","Data":"95b98b8e37bf2a9c3c885306d982608d98a7d2ff767e5dd183260b5fe9652063"}
Jul 31 13:00:52.849441 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.849421    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:00:52.850997 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.850816    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm" event={"ID":"f0e213fb-c3ba-4a80-97ca-9022fea600ab","Type":"ContainerStarted","Data":"88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a"}
Jul 31 13:00:52.850997 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.850857    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm" event={"ID":"f0e213fb-c3ba-4a80-97ca-9022fea600ab","Type":"ContainerStarted","Data":"1055fd3d934e18968207f78b2b8b160f7aea0e42bc5e1c826c6c16921593f532"}
Jul 31 13:00:52.850997 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.850887    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:00:52.865411 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.865373    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c" podStartSLOduration=2.442522904 podStartE2EDuration="2.865359986s" totalImagesPullingTime="422.837082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:00:52.86467157 +0000 UTC m=+2740.851893110" watchObservedRunningTime="2026-07-31 13:00:52.865359986 +0000 UTC m=+2740.852581526"
Jul 31 13:00:52.882503 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.882451    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm" podStartSLOduration=1.8824314549999999 podStartE2EDuration="1.882431455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:00:52.879669112 +0000 UTC m=+2740.866890652" watchObservedRunningTime="2026-07-31 13:00:52.882431455 +0000 UTC m=+2740.869652996"
Jul 31 13:00:52.897430 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:52.897388    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2" podStartSLOduration=1.8973737229999998 podStartE2EDuration="1.897373723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:00:52.895354538 +0000 UTC m=+2740.882576082" watchObservedRunningTime="2026-07-31 13:00:52.897373723 +0000 UTC m=+2740.884595262"
Jul 31 13:00:58.864492 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:00:58.864465    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:01:02.907599 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:02.907567    2586 generic.go:356] "Generic (PLEG): container finished" podID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerID="4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22" exitCode=0
Jul 31 13:01:02.908115 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:02.907636    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" event={"ID":"e9e26fe4-e454-4c99-8bec-90e0852042b3","Type":"ContainerDied","Data":"4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22"}
Jul 31 13:01:02.908115 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:02.908030    2586 scope.go:117] "RemoveContainer" containerID="4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22"
Jul 31 13:01:02.909310 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:02.909278    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:01:02.909397 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:02.909373    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:01:02.909438 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:02.909425    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:01:03.825375 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:03.825342    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:01:03.860040 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:03.860004    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:01:03.861040 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:03.861014    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:01:03.915230 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:03.915173    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" event={"ID":"e9e26fe4-e454-4c99-8bec-90e0852042b3","Type":"ContainerStarted","Data":"33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d"}
Jul 31 13:01:04.753401 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:04.753363    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:01:04.755019 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:04.754992    2586 patch_prober.go:28] interesting pod/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 31 13:01:04.755229 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:04.755203    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 31 13:01:05.752905 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:05.752860    2586 patch_prober.go:28] interesting pod/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 31 13:01:05.753421 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:05.752929    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 31 13:01:06.753066 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:06.753022    2586 patch_prober.go:28] interesting pod/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 31 13:01:06.753596 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:06.753088    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 31 13:01:07.753198 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:07.753153    2586 patch_prober.go:28] interesting pod/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 31 13:01:07.753719 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:07.753240    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 31 13:01:08.753649 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:08.753608    2586 patch_prober.go:28] interesting pod/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 31 13:01:08.754039 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:08.753667    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 31 13:01:09.757069 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:09.757039    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:01:09.757482 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:09.757418    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:01:09.758145 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:09.758127    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:01:20.099882 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.099852    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"]
Jul 31 13:01:20.102548 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.102524    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.105354 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.105330    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b21077-serving-cert\""
Jul 31 13:01:20.114366 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.114341    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"]
Jul 31 13:01:20.242823 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.242787    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zrrx\" (UniqueName: \"kubernetes.io/projected/da8a23f4-e070-47b1-b61d-9ae30893b8ec-kube-api-access-6zrrx\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.243020 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.242850    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a23f4-e070-47b1-b61d-9ae30893b8ec-maas-api-tls\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.243020 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.242960    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a23f4-e070-47b1-b61d-9ae30893b8ec-openshift-service-ca\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.344438 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.344404    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a23f4-e070-47b1-b61d-9ae30893b8ec-maas-api-tls\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.344651 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.344475    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a23f4-e070-47b1-b61d-9ae30893b8ec-openshift-service-ca\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.344651 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.344522    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zrrx\" (UniqueName: \"kubernetes.io/projected/da8a23f4-e070-47b1-b61d-9ae30893b8ec-kube-api-access-6zrrx\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.344651 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:01:20.344588    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-serving-cert: secret "maas-api-e2e-api-b-b21077-serving-cert" not found
Jul 31 13:01:20.344825 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:01:20.344658    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8a23f4-e070-47b1-b61d-9ae30893b8ec-maas-api-tls podName:da8a23f4-e070-47b1-b61d-9ae30893b8ec nodeName:}" failed. No retries permitted until 2026-07-31 13:01:20.844637222 +0000 UTC m=+2768.831858740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da8a23f4-e070-47b1-b61d-9ae30893b8ec-maas-api-tls") pod "maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr" (UID: "da8a23f4-e070-47b1-b61d-9ae30893b8ec") : secret "maas-api-e2e-api-b-b21077-serving-cert" not found
Jul 31 13:01:20.345062 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.345043    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a23f4-e070-47b1-b61d-9ae30893b8ec-openshift-service-ca\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.354566 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.354514    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zrrx\" (UniqueName: \"kubernetes.io/projected/da8a23f4-e070-47b1-b61d-9ae30893b8ec-kube-api-access-6zrrx\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.809774 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.809745    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"]
Jul 31 13:01:20.813063 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.813049    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:01:20.815655 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.815638    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b21077\""
Jul 31 13:01:20.815733 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.815676    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b21077-dockercfg-gbsn4\""
Jul 31 13:01:20.835298 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.835200    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"]
Jul 31 13:01:20.850542 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.850474    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a23f4-e070-47b1-b61d-9ae30893b8ec-maas-api-tls\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.853855 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.853828    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a23f4-e070-47b1-b61d-9ae30893b8ec-maas-api-tls\") pod \"maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:20.951199 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.951160    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4gg\" (UniqueName: \"kubernetes.io/projected/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-kube-api-access-sv4gg\") pod \"payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5\" (UID: \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:01:20.951404 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:20.951210    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5\" (UID: \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:01:21.014822 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:21.014791    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:21.052193 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:21.052163    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4gg\" (UniqueName: \"kubernetes.io/projected/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-kube-api-access-sv4gg\") pod \"payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5\" (UID: \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:01:21.052338 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:21.052210    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5\" (UID: \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:01:21.052872 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:21.052849    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5\" (UID: \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:01:21.063660 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:21.063590    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4gg\" (UniqueName: \"kubernetes.io/projected/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-kube-api-access-sv4gg\") pod \"payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5\" (UID: \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:01:21.124691 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:21.124657    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:01:21.150810 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:21.150780    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"]
Jul 31 13:01:21.152315 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:01:21.152287    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8a23f4_e070_47b1_b61d_9ae30893b8ec.slice/crio-7665f7356b9cde761520a87e527b7bed8e7ce48db714891c3fd4a3514210c417 WatchSource:0}: Error finding container 7665f7356b9cde761520a87e527b7bed8e7ce48db714891c3fd4a3514210c417: Status 404 returned error can't find the container with id 7665f7356b9cde761520a87e527b7bed8e7ce48db714891c3fd4a3514210c417
Jul 31 13:01:21.264492 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:21.264466    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"]
Jul 31 13:01:21.266407 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:01:21.266380    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66eacdb6_ebfe_4fc2_a9e4_1e704717c5b3.slice/crio-e44acf1a087c767db04787ade36991c41d2eac7916f194ce272f043c524e6926 WatchSource:0}: Error finding container e44acf1a087c767db04787ade36991c41d2eac7916f194ce272f043c524e6926: Status 404 returned error can't find the container with id e44acf1a087c767db04787ade36991c41d2eac7916f194ce272f043c524e6926
Jul 31 13:01:22.016461 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:22.016424    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5" event={"ID":"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3","Type":"ContainerStarted","Data":"26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91"}
Jul 31 13:01:22.016667 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:22.016485    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5" event={"ID":"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3","Type":"ContainerStarted","Data":"e44acf1a087c767db04787ade36991c41d2eac7916f194ce272f043c524e6926"}
Jul 31 13:01:22.016667 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:22.016529    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:01:22.017973 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:22.017950    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr" event={"ID":"da8a23f4-e070-47b1-b61d-9ae30893b8ec","Type":"ContainerStarted","Data":"2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639"}
Jul 31 13:01:22.018081 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:22.017986    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr" event={"ID":"da8a23f4-e070-47b1-b61d-9ae30893b8ec","Type":"ContainerStarted","Data":"7665f7356b9cde761520a87e527b7bed8e7ce48db714891c3fd4a3514210c417"}
Jul 31 13:01:22.018121 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:22.018079    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:22.036080 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:22.036043    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5" podStartSLOduration=2.036028938 podStartE2EDuration="2.036028938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:01:22.034608703 +0000 UTC m=+2770.021830244" watchObservedRunningTime="2026-07-31 13:01:22.036028938 +0000 UTC m=+2770.023250478"
Jul 31 13:01:22.055595 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:22.055555    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr" podStartSLOduration=1.636818254 podStartE2EDuration="2.055543929s" totalImagesPullingTime="418.725675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:01:22.053417139 +0000 UTC m=+2770.040638692" watchObservedRunningTime="2026-07-31 13:01:22.055543929 +0000 UTC m=+2770.042765468"
Jul 31 13:01:28.028559 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:28.028531    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:01:33.025186 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:01:33.025156    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:02:02.178073 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.177980    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"]
Jul 31 13:02:02.178649 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.178316    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c" podUID="a0352937-a4e7-4ee7-8f57-1078a8ae5d9e" containerName="maas-api" containerID="cri-o://99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908" gracePeriod=30
Jul 31 13:02:02.228900 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.228868    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"]
Jul 31 13:02:02.229167 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.229114    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2" podUID="d0e46e26-ffc8-4aef-958d-6ce4df70f878" containerName="payload-processing" containerID="cri-o://5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af" gracePeriod=30
Jul 31 13:02:02.233632 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.233607    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"]
Jul 31 13:02:02.233845 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.233818    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm" podUID="f0e213fb-c3ba-4a80-97ca-9022fea600ab" containerName="payload-pre-processing" containerID="cri-o://88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a" gracePeriod=30
Jul 31 13:02:02.498625 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.498597    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:02:02.542019 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.541996    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:02:02.545800 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.545778    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:02:02.614458 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.614432    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-openshift-service-ca\") pod \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") "
Jul 31 13:02:02.614458 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.614467    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmv4c\" (UniqueName: \"kubernetes.io/projected/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-kube-api-access-gmv4c\") pod \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") "
Jul 31 13:02:02.614663 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.614562    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-maas-api-tls\") pod \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\" (UID: \"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e\") "
Jul 31 13:02:02.614663 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.614580    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e46e26-ffc8-4aef-958d-6ce4df70f878-plugins-config-volume\") pod \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\" (UID: \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\") "
Jul 31 13:02:02.614663 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.614603    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtv75\" (UniqueName: \"kubernetes.io/projected/d0e46e26-ffc8-4aef-958d-6ce4df70f878-kube-api-access-qtv75\") pod \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\" (UID: \"d0e46e26-ffc8-4aef-958d-6ce4df70f878\") "
Jul 31 13:02:02.615034 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.614923    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0352937-a4e7-4ee7-8f57-1078a8ae5d9e" (UID: "a0352937-a4e7-4ee7-8f57-1078a8ae5d9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:02:02.615034 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.614995    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e46e26-ffc8-4aef-958d-6ce4df70f878-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0e46e26-ffc8-4aef-958d-6ce4df70f878" (UID: "d0e46e26-ffc8-4aef-958d-6ce4df70f878"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:02:02.616771 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.616747    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-kube-api-access-gmv4c" (OuterVolumeSpecName: "kube-api-access-gmv4c") pod "a0352937-a4e7-4ee7-8f57-1078a8ae5d9e" (UID: "a0352937-a4e7-4ee7-8f57-1078a8ae5d9e"). InnerVolumeSpecName "kube-api-access-gmv4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:02:02.616865 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.616781    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e46e26-ffc8-4aef-958d-6ce4df70f878-kube-api-access-qtv75" (OuterVolumeSpecName: "kube-api-access-qtv75") pod "d0e46e26-ffc8-4aef-958d-6ce4df70f878" (UID: "d0e46e26-ffc8-4aef-958d-6ce4df70f878"). InnerVolumeSpecName "kube-api-access-qtv75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:02:02.616865 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.616797    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0352937-a4e7-4ee7-8f57-1078a8ae5d9e" (UID: "a0352937-a4e7-4ee7-8f57-1078a8ae5d9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 13:02:02.716110 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.716021    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e213fb-c3ba-4a80-97ca-9022fea600ab-plugins-config-volume\") pod \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\" (UID: \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\") "
Jul 31 13:02:02.716257 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.716125    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj8vb\" (UniqueName: \"kubernetes.io/projected/f0e213fb-c3ba-4a80-97ca-9022fea600ab-kube-api-access-sj8vb\") pod \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\" (UID: \"f0e213fb-c3ba-4a80-97ca-9022fea600ab\") "
Jul 31 13:02:02.716488 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.716463    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e213fb-c3ba-4a80-97ca-9022fea600ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0e213fb-c3ba-4a80-97ca-9022fea600ab" (UID: "f0e213fb-c3ba-4a80-97ca-9022fea600ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:02:02.716530 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.716478    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:02.716530 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.716514    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e46e26-ffc8-4aef-958d-6ce4df70f878-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:02.716530 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.716526    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtv75\" (UniqueName: \"kubernetes.io/projected/d0e46e26-ffc8-4aef-958d-6ce4df70f878-kube-api-access-qtv75\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:02.716640 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.716537    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:02.716640 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.716545    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmv4c\" (UniqueName: \"kubernetes.io/projected/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e-kube-api-access-gmv4c\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:02.718096 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.718076    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e213fb-c3ba-4a80-97ca-9022fea600ab-kube-api-access-sj8vb" (OuterVolumeSpecName: "kube-api-access-sj8vb") pod "f0e213fb-c3ba-4a80-97ca-9022fea600ab" (UID: "f0e213fb-c3ba-4a80-97ca-9022fea600ab"). InnerVolumeSpecName "kube-api-access-sj8vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:02:02.817522 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.817501    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj8vb\" (UniqueName: \"kubernetes.io/projected/f0e213fb-c3ba-4a80-97ca-9022fea600ab-kube-api-access-sj8vb\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:02.817608 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:02.817529    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e213fb-c3ba-4a80-97ca-9022fea600ab-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:03.216164 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.216131    2586 generic.go:356] "Generic (PLEG): container finished" podID="a0352937-a4e7-4ee7-8f57-1078a8ae5d9e" containerID="99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908" exitCode=0
Jul 31 13:02:03.216619 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.216174    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"
Jul 31 13:02:03.216619 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.216224    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c" event={"ID":"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e","Type":"ContainerDied","Data":"99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908"}
Jul 31 13:02:03.216619 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.216303    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c" event={"ID":"a0352937-a4e7-4ee7-8f57-1078a8ae5d9e","Type":"ContainerDied","Data":"ca360bcf5a15846a128521bd5467e6547b874e3a173ea797ef4b4fdc287aa51e"}
Jul 31 13:02:03.216619 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.216339    2586 scope.go:117] "RemoveContainer" containerID="99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908"
Jul 31 13:02:03.217736 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.217462    2586 generic.go:356] "Generic (PLEG): container finished" podID="d0e46e26-ffc8-4aef-958d-6ce4df70f878" containerID="5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af" exitCode=0
Jul 31 13:02:03.217736 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.217509    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"
Jul 31 13:02:03.217736 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.217524    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2" event={"ID":"d0e46e26-ffc8-4aef-958d-6ce4df70f878","Type":"ContainerDied","Data":"5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af"}
Jul 31 13:02:03.217736 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.217561    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2" event={"ID":"d0e46e26-ffc8-4aef-958d-6ce4df70f878","Type":"ContainerDied","Data":"95b98b8e37bf2a9c3c885306d982608d98a7d2ff767e5dd183260b5fe9652063"}
Jul 31 13:02:03.218907 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.218878    2586 generic.go:356] "Generic (PLEG): container finished" podID="f0e213fb-c3ba-4a80-97ca-9022fea600ab" containerID="88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a" exitCode=0
Jul 31 13:02:03.219013 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.218923    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"
Jul 31 13:02:03.219013 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.218967    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm" event={"ID":"f0e213fb-c3ba-4a80-97ca-9022fea600ab","Type":"ContainerDied","Data":"88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a"}
Jul 31 13:02:03.219104 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.219015    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm" event={"ID":"f0e213fb-c3ba-4a80-97ca-9022fea600ab","Type":"ContainerDied","Data":"1055fd3d934e18968207f78b2b8b160f7aea0e42bc5e1c826c6c16921593f532"}
Jul 31 13:02:03.225521 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.225484    2586 scope.go:117] "RemoveContainer" containerID="99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908"
Jul 31 13:02:03.225779 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:03.225752    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908\": container with ID starting with 99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908 not found: ID does not exist" containerID="99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908"
Jul 31 13:02:03.225897 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.225788    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908"} err="failed to get container status \"99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908\": rpc error: code = NotFound desc = could not find container \"99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908\": container with ID starting with 99ce37a4314dad5de26a1cf542782d1cb85863b3868d3698fcd8e5eb71768908 not found: ID does not exist"
Jul 31 13:02:03.225897 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.225812    2586 scope.go:117] "RemoveContainer" containerID="5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af"
Jul 31 13:02:03.234607 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.234589    2586 scope.go:117] "RemoveContainer" containerID="5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af"
Jul 31 13:02:03.234859 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:03.234840    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af\": container with ID starting with 5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af not found: ID does not exist" containerID="5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af"
Jul 31 13:02:03.234913 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.234867    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af"} err="failed to get container status \"5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af\": rpc error: code = NotFound desc = could not find container \"5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af\": container with ID starting with 5bd26a9b8b1183fca3e3135bc215c9ecb98e28cc25c33f7aa335b4c6bc13c5af not found: ID does not exist"
Jul 31 13:02:03.234913 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.234890    2586 scope.go:117] "RemoveContainer" containerID="88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a"
Jul 31 13:02:03.242300 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.242260    2586 scope.go:117] "RemoveContainer" containerID="88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a"
Jul 31 13:02:03.242544 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:03.242528    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a\": container with ID starting with 88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a not found: ID does not exist" containerID="88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a"
Jul 31 13:02:03.242593 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.242550    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a"} err="failed to get container status \"88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a\": rpc error: code = NotFound desc = could not find container \"88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a\": container with ID starting with 88f1968001fa2c9506020f66d189163f619b5ecb04115eea8525083ddf05435a not found: ID does not exist"
Jul 31 13:02:03.244181 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.244163    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"]
Jul 31 13:02:03.248435 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.248418    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c802-6cb8c9b48d-zjzkm"]
Jul 31 13:02:03.259353 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.259333    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"]
Jul 31 13:02:03.263632 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.263614    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c802-67fcd7c996-wxp7c"]
Jul 31 13:02:03.274967 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.274949    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"]
Jul 31 13:02:03.276712 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:03.276694    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c802-555b9ccb74-d2fn2"]
Jul 31 13:02:04.484710 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:04.484670    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0352937-a4e7-4ee7-8f57-1078a8ae5d9e" path="/var/lib/kubelet/pods/a0352937-a4e7-4ee7-8f57-1078a8ae5d9e/volumes"
Jul 31 13:02:04.485212 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:04.485142    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e46e26-ffc8-4aef-958d-6ce4df70f878" path="/var/lib/kubelet/pods/d0e46e26-ffc8-4aef-958d-6ce4df70f878/volumes"
Jul 31 13:02:04.485536 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:04.485518    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e213fb-c3ba-4a80-97ca-9022fea600ab" path="/var/lib/kubelet/pods/f0e213fb-c3ba-4a80-97ca-9022fea600ab/volumes"
Jul 31 13:02:08.855030 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:08.855000    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"]
Jul 31 13:02:08.855473 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:08.855212    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy" containerID="cri-o://33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d" gracePeriod=30
Jul 31 13:02:09.759572 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:09.759537    2586 patch_prober.go:28] interesting pod/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 13:02:09.759751 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:09.759602    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 13:02:14.123506 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.123444    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:02:14.213838 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.213806    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-token\") pod \"e9e26fe4-e454-4c99-8bec-90e0852042b3\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") "
Jul 31 13:02:14.213988 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.213860    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-envoy\") pod \"e9e26fe4-e454-4c99-8bec-90e0852042b3\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") "
Jul 31 13:02:14.213988 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.213875    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-data\") pod \"e9e26fe4-e454-4c99-8bec-90e0852042b3\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") "
Jul 31 13:02:14.213988 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.213947    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqrlx\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-kube-api-access-sqrlx\") pod \"e9e26fe4-e454-4c99-8bec-90e0852042b3\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") "
Jul 31 13:02:14.213988 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.213979    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-podinfo\") pod \"e9e26fe4-e454-4c99-8bec-90e0852042b3\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") "
Jul 31 13:02:14.214182 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214013    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-certs\") pod \"e9e26fe4-e454-4c99-8bec-90e0852042b3\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") "
Jul 31 13:02:14.214182 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214037    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e26fe4-e454-4c99-8bec-90e0852042b3-istiod-ca-cert\") pod \"e9e26fe4-e454-4c99-8bec-90e0852042b3\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") "
Jul 31 13:02:14.214182 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214061    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-credential-socket\") pod \"e9e26fe4-e454-4c99-8bec-90e0852042b3\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") "
Jul 31 13:02:14.214182 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214089    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-socket\") pod \"e9e26fe4-e454-4c99-8bec-90e0852042b3\" (UID: \"e9e26fe4-e454-4c99-8bec-90e0852042b3\") "
Jul 31 13:02:14.214414 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214219    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9e26fe4-e454-4c99-8bec-90e0852042b3" (UID: "e9e26fe4-e454-4c99-8bec-90e0852042b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:02:14.214414 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214326    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9e26fe4-e454-4c99-8bec-90e0852042b3" (UID: "e9e26fe4-e454-4c99-8bec-90e0852042b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:02:14.214512 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214444    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:14.214512 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214463    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:14.214512 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214453    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9e26fe4-e454-4c99-8bec-90e0852042b3" (UID: "e9e26fe4-e454-4c99-8bec-90e0852042b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:02:14.214625 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214510    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9e26fe4-e454-4c99-8bec-90e0852042b3" (UID: "e9e26fe4-e454-4c99-8bec-90e0852042b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:02:14.214625 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.214501    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e26fe4-e454-4c99-8bec-90e0852042b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9e26fe4-e454-4c99-8bec-90e0852042b3" (UID: "e9e26fe4-e454-4c99-8bec-90e0852042b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:02:14.216130 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.216099    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9e26fe4-e454-4c99-8bec-90e0852042b3" (UID: "e9e26fe4-e454-4c99-8bec-90e0852042b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:02:14.216397 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.216368    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-kube-api-access-sqrlx" (OuterVolumeSpecName: "kube-api-access-sqrlx") pod "e9e26fe4-e454-4c99-8bec-90e0852042b3" (UID: "e9e26fe4-e454-4c99-8bec-90e0852042b3"). InnerVolumeSpecName "kube-api-access-sqrlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:02:14.216511 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.216449    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9e26fe4-e454-4c99-8bec-90e0852042b3" (UID: "e9e26fe4-e454-4c99-8bec-90e0852042b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 13:02:14.216511 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.216465    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9e26fe4-e454-4c99-8bec-90e0852042b3" (UID: "e9e26fe4-e454-4c99-8bec-90e0852042b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:02:14.281945 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.281914    2586 generic.go:356] "Generic (PLEG): container finished" podID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerID="33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d" exitCode=0
Jul 31 13:02:14.282047 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.281984    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"
Jul 31 13:02:14.282047 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.281999    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" event={"ID":"e9e26fe4-e454-4c99-8bec-90e0852042b3","Type":"ContainerDied","Data":"33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d"}
Jul 31 13:02:14.282128 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.282079    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct" event={"ID":"e9e26fe4-e454-4c99-8bec-90e0852042b3","Type":"ContainerDied","Data":"b74e00424d21b7521a82db10f372cb02eaa755df4fbfa7d4dd962052b39fbd36"}
Jul 31 13:02:14.282128 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.282101    2586 scope.go:117] "RemoveContainer" containerID="33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d"
Jul 31 13:02:14.290723 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.290708    2586 scope.go:117] "RemoveContainer" containerID="4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22"
Jul 31 13:02:14.298481 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.298466    2586 scope.go:117] "RemoveContainer" containerID="33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d"
Jul 31 13:02:14.298712 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:14.298693    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d\": container with ID starting with 33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d not found: ID does not exist" containerID="33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d"
Jul 31 13:02:14.298766 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.298718    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d"} err="failed to get container status \"33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d\": rpc error: code = NotFound desc = could not find container \"33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d\": container with ID starting with 33e358f05b20673b5c466777b2d3dc54104ae6feda8d782795ed1cffb5a5709d not found: ID does not exist"
Jul 31 13:02:14.298766 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.298735    2586 scope.go:117] "RemoveContainer" containerID="4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22"
Jul 31 13:02:14.298945 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:14.298929    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22\": container with ID starting with 4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22 not found: ID does not exist" containerID="4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22"
Jul 31 13:02:14.298989 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.298948    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22"} err="failed to get container status \"4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22\": rpc error: code = NotFound desc = could not find container \"4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22\": container with ID starting with 4746d6a1adeb7a9c7e50d08e06fa5ee0724c3073efcb430c3a8942d518b85c22 not found: ID does not exist"
Jul 31 13:02:14.304773 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.304753    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"]
Jul 31 13:02:14.308668 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.308648    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b0c802-openshift-default-fdd8d965d-kbcct"]
Jul 31 13:02:14.315426 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.315411    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:14.315483 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.315428    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:14.315483 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.315437    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqrlx\" (UniqueName: \"kubernetes.io/projected/e9e26fe4-e454-4c99-8bec-90e0852042b3-kube-api-access-sqrlx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:14.315483 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.315448    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e26fe4-e454-4c99-8bec-90e0852042b3-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:14.315483 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.315457    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e26fe4-e454-4c99-8bec-90e0852042b3-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:14.315483 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.315465    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:14.315483 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.315473    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e26fe4-e454-4c99-8bec-90e0852042b3-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:14.484765 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:14.484733    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" path="/var/lib/kubelet/pods/e9e26fe4-e454-4c99-8bec-90e0852042b3/volumes"
Jul 31 13:02:24.608715 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.608680    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"]
Jul 31 13:02:24.609156 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.608905    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr" podUID="da8a23f4-e070-47b1-b61d-9ae30893b8ec" containerName="maas-api" containerID="cri-o://2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639" gracePeriod=30
Jul 31 13:02:24.729517 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.729485    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"]
Jul 31 13:02:24.729753 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.729727    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5" podUID="66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3" containerName="payload-pre-processing" containerID="cri-o://26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91" gracePeriod=30
Jul 31 13:02:24.870603 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.870544    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:02:24.904043 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.904013    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zrrx\" (UniqueName: \"kubernetes.io/projected/da8a23f4-e070-47b1-b61d-9ae30893b8ec-kube-api-access-6zrrx\") pod \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") "
Jul 31 13:02:24.904208 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.904068    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a23f4-e070-47b1-b61d-9ae30893b8ec-maas-api-tls\") pod \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") "
Jul 31 13:02:24.904208 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.904116    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a23f4-e070-47b1-b61d-9ae30893b8ec-openshift-service-ca\") pod \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\" (UID: \"da8a23f4-e070-47b1-b61d-9ae30893b8ec\") "
Jul 31 13:02:24.904561 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.904527    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8a23f4-e070-47b1-b61d-9ae30893b8ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da8a23f4-e070-47b1-b61d-9ae30893b8ec" (UID: "da8a23f4-e070-47b1-b61d-9ae30893b8ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:02:24.906922 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.906893    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8a23f4-e070-47b1-b61d-9ae30893b8ec-kube-api-access-6zrrx" (OuterVolumeSpecName: "kube-api-access-6zrrx") pod "da8a23f4-e070-47b1-b61d-9ae30893b8ec" (UID: "da8a23f4-e070-47b1-b61d-9ae30893b8ec"). InnerVolumeSpecName "kube-api-access-6zrrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:02:24.907052 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.906991    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8a23f4-e070-47b1-b61d-9ae30893b8ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da8a23f4-e070-47b1-b61d-9ae30893b8ec" (UID: "da8a23f4-e070-47b1-b61d-9ae30893b8ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 13:02:24.978176 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:24.978152    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:02:25.005318 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.005256    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-plugins-config-volume\") pod \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\" (UID: \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\") "
Jul 31 13:02:25.005458 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.005404    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv4gg\" (UniqueName: \"kubernetes.io/projected/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-kube-api-access-sv4gg\") pod \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\" (UID: \"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3\") "
Jul 31 13:02:25.005606 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.005585    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3" (UID: "66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:02:25.005706 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.005691    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a23f4-e070-47b1-b61d-9ae30893b8ec-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:25.005768 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.005707    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:25.005768 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.005718    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a23f4-e070-47b1-b61d-9ae30893b8ec-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:25.005768 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.005726    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zrrx\" (UniqueName: \"kubernetes.io/projected/da8a23f4-e070-47b1-b61d-9ae30893b8ec-kube-api-access-6zrrx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:25.007329 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.007312    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-kube-api-access-sv4gg" (OuterVolumeSpecName: "kube-api-access-sv4gg") pod "66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3" (UID: "66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3"). InnerVolumeSpecName "kube-api-access-sv4gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:02:25.106664 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.106632    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv4gg\" (UniqueName: \"kubernetes.io/projected/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3-kube-api-access-sv4gg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:02:25.337078 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.337041    2586 generic.go:356] "Generic (PLEG): container finished" podID="66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3" containerID="26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91" exitCode=0
Jul 31 13:02:25.337244 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.337085    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"
Jul 31 13:02:25.337244 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.337125    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5" event={"ID":"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3","Type":"ContainerDied","Data":"26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91"}
Jul 31 13:02:25.337244 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.337185    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5" event={"ID":"66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3","Type":"ContainerDied","Data":"e44acf1a087c767db04787ade36991c41d2eac7916f194ce272f043c524e6926"}
Jul 31 13:02:25.337244 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.337211    2586 scope.go:117] "RemoveContainer" containerID="26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91"
Jul 31 13:02:25.338477 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.338451    2586 generic.go:356] "Generic (PLEG): container finished" podID="da8a23f4-e070-47b1-b61d-9ae30893b8ec" containerID="2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639" exitCode=0
Jul 31 13:02:25.338599 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.338482    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr" event={"ID":"da8a23f4-e070-47b1-b61d-9ae30893b8ec","Type":"ContainerDied","Data":"2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639"}
Jul 31 13:02:25.338599 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.338532    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr" event={"ID":"da8a23f4-e070-47b1-b61d-9ae30893b8ec","Type":"ContainerDied","Data":"7665f7356b9cde761520a87e527b7bed8e7ce48db714891c3fd4a3514210c417"}
Jul 31 13:02:25.338599 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.338491    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"
Jul 31 13:02:25.345797 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.345777    2586 scope.go:117] "RemoveContainer" containerID="26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91"
Jul 31 13:02:25.346065 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:25.346042    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91\": container with ID starting with 26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91 not found: ID does not exist" containerID="26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91"
Jul 31 13:02:25.346145 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.346076    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91"} err="failed to get container status \"26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91\": rpc error: code = NotFound desc = could not find container \"26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91\": container with ID starting with 26fe835f70475cbed8970258f3dc89f78b24e8a9c090422402761641626afd91 not found: ID does not exist"
Jul 31 13:02:25.346145 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.346101    2586 scope.go:117] "RemoveContainer" containerID="2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639"
Jul 31 13:02:25.354051 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.354034    2586 scope.go:117] "RemoveContainer" containerID="2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639"
Jul 31 13:02:25.354312 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:25.354293    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639\": container with ID starting with 2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639 not found: ID does not exist" containerID="2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639"
Jul 31 13:02:25.354377 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.354320    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639"} err="failed to get container status \"2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639\": rpc error: code = NotFound desc = could not find container \"2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639\": container with ID starting with 2569ddd7ec51482d863757efadb82a2ba5edc2097f2750b07e586d23161f5639 not found: ID does not exist"
Jul 31 13:02:25.362180 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.362159    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"]
Jul 31 13:02:25.369910 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.369891    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b21077-5b7d6646b4-mgsc5"]
Jul 31 13:02:25.379214 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.379193    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"]
Jul 31 13:02:25.383624 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:25.383607    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b21077-6d64c5fcdc-mg8xr"]
Jul 31 13:02:26.485431 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:26.485389    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3" path="/var/lib/kubelet/pods/66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3/volumes"
Jul 31 13:02:26.485959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:26.485938    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da8a23f4-e070-47b1-b61d-9ae30893b8ec" path="/var/lib/kubelet/pods/da8a23f4-e070-47b1-b61d-9ae30893b8ec/volumes"
Jul 31 13:02:32.464002 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.463965    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"]
Jul 31 13:02:32.464505 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464489    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy"
Jul 31 13:02:32.464579 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464506    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy"
Jul 31 13:02:32.464579 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464522    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8a23f4-e070-47b1-b61d-9ae30893b8ec" containerName="maas-api"
Jul 31 13:02:32.464579 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464530    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8a23f4-e070-47b1-b61d-9ae30893b8ec" containerName="maas-api"
Jul 31 13:02:32.464579 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464552    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0352937-a4e7-4ee7-8f57-1078a8ae5d9e" containerName="maas-api"
Jul 31 13:02:32.464579 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464560    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0352937-a4e7-4ee7-8f57-1078a8ae5d9e" containerName="maas-api"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464584    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e213fb-c3ba-4a80-97ca-9022fea600ab" containerName="payload-pre-processing"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464593    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e213fb-c3ba-4a80-97ca-9022fea600ab" containerName="payload-pre-processing"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464608    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3" containerName="payload-pre-processing"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464617    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3" containerName="payload-pre-processing"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464633    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e46e26-ffc8-4aef-958d-6ce4df70f878" containerName="payload-processing"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464641    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e46e26-ffc8-4aef-958d-6ce4df70f878" containerName="payload-processing"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464720    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0352937-a4e7-4ee7-8f57-1078a8ae5d9e" containerName="maas-api"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464731    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464740    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e213fb-c3ba-4a80-97ca-9022fea600ab" containerName="payload-pre-processing"
Jul 31 13:02:32.464755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464749    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="da8a23f4-e070-47b1-b61d-9ae30893b8ec" containerName="maas-api"
Jul 31 13:02:32.465070 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464759    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e46e26-ffc8-4aef-958d-6ce4df70f878" containerName="payload-processing"
Jul 31 13:02:32.465070 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.464768    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="66eacdb6-ebfe-4fc2-a9e4-1e704717c5b3" containerName="payload-pre-processing"
Jul 31 13:02:32.469640 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.469616    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.473117 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.473096    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-96b3e0-openshift-default-dockercfg-zzdv4\""
Jul 31 13:02:32.486374 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.486346    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"]
Jul 31 13:02:32.570900 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.570860    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-data\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.571090 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.570908    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-podinfo\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.571090 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.570945    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-socket\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.571090 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.570969    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4af621-bf34-49a3-8870-ecbd873d50b2-istiod-ca-cert\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.571252 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.571093    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-certs\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.571252 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.571151    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-envoy\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.571252 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.571180    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-token\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.571252 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.571204    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rkv\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-kube-api-access-d9rkv\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.571493 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.571327    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-credential-socket\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.672791 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.672755    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-envoy\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.672967 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.672803    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-token\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.672967 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.672928    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rkv\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-kube-api-access-d9rkv\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673107 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673017    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-credential-socket\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673107 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673078    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-data\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673217 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673107    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-podinfo\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673217 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673161    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-socket\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673217 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673197    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4af621-bf34-49a3-8870-ecbd873d50b2-istiod-ca-cert\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673403 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673241    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-certs\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673500 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673474    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-data\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673689 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673505    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-socket\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673775 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673592    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-certs\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673853 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673834    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4af621-bf34-49a3-8870-ecbd873d50b2-istiod-ca-cert\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.673912 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.673869    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-credential-socket\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.675148 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.675125    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-envoy\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.675423 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.675407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-podinfo\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.681410 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.681388    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rkv\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-kube-api-access-d9rkv\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.681799 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.681778    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-token\") pod \"e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") " pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.782013 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.781936    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:32.917984 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.917956    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"]
Jul 31 13:02:32.921292 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:02:32.921247    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4af621_bf34_49a3_8870_ecbd873d50b2.slice/crio-992c7944f90bbf21b36b1a3e10000e39aaefbe99306e851d8713710ee28b795d WatchSource:0}: Error finding container 992c7944f90bbf21b36b1a3e10000e39aaefbe99306e851d8713710ee28b795d: Status 404 returned error can't find the container with id 992c7944f90bbf21b36b1a3e10000e39aaefbe99306e851d8713710ee28b795d
Jul 31 13:02:32.923684 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.923653    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:02:32.923773 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.923719    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:02:32.923773 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:32.923747    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:02:33.378569 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:33.378532    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" event={"ID":"ba4af621-bf34-49a3-8870-ecbd873d50b2","Type":"ContainerStarted","Data":"bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5"}
Jul 31 13:02:33.378802 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:33.378594    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" event={"ID":"ba4af621-bf34-49a3-8870-ecbd873d50b2","Type":"ContainerStarted","Data":"992c7944f90bbf21b36b1a3e10000e39aaefbe99306e851d8713710ee28b795d"}
Jul 31 13:02:33.399395 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:33.399351    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" podStartSLOduration=1.3993360400000001 podStartE2EDuration="1.39933604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:02:33.397347588 +0000 UTC m=+2841.384569125" watchObservedRunningTime="2026-07-31 13:02:33.39933604 +0000 UTC m=+2841.386557580"
Jul 31 13:02:33.783042 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:33.783014    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:33.788403 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:33.788381    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:34.383605 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:34.383575    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:34.384518 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:34.384497    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:36.398899 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.398867    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"]
Jul 31 13:02:36.399431 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.399414    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy"
Jul 31 13:02:36.399515 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.399433    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy"
Jul 31 13:02:36.399577 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.399552    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e26fe4-e454-4c99-8bec-90e0852042b3" containerName="istio-proxy"
Jul 31 13:02:36.402677 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.402655    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:36.405065 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.405028    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-96b3e0-serving-cert\""
Jul 31 13:02:36.412113 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.412083    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"]
Jul 31 13:02:36.506241 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.506205    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fd4400-bb38-4f3a-9eb7-4119889c64fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:36.506241 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.506242    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjkgt\" (UniqueName: \"kubernetes.io/projected/01fd4400-bb38-4f3a-9eb7-4119889c64fa-kube-api-access-bjkgt\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:36.506542 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.506403    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:36.607529 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.607500    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fd4400-bb38-4f3a-9eb7-4119889c64fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:36.607529 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.607530    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjkgt\" (UniqueName: \"kubernetes.io/projected/01fd4400-bb38-4f3a-9eb7-4119889c64fa-kube-api-access-bjkgt\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:36.607780 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.607566    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:36.607780 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:36.607683    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-serving-cert: secret "maas-api-e2e-shared-a-96b3e0-serving-cert" not found
Jul 31 13:02:36.607780 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:36.607745    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls podName:01fd4400-bb38-4f3a-9eb7-4119889c64fa nodeName:}" failed. No retries permitted until 2026-07-31 13:02:37.107730116 +0000 UTC m=+2845.094951634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls") pod "maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2" (UID: "01fd4400-bb38-4f3a-9eb7-4119889c64fa") : secret "maas-api-e2e-shared-a-96b3e0-serving-cert" not found
Jul 31 13:02:36.608248 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.608225    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fd4400-bb38-4f3a-9eb7-4119889c64fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:36.616167 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:36.616142    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjkgt\" (UniqueName: \"kubernetes.io/projected/01fd4400-bb38-4f3a-9eb7-4119889c64fa-kube-api-access-bjkgt\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:37.068725 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.068687    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"]
Jul 31 13:02:37.072401 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.072384    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:37.074757 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.074735    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-96b3e0\""
Jul 31 13:02:37.074881 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.074812    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-96b3e0-dockercfg-jcjpd\""
Jul 31 13:02:37.082399 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.082377    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"]
Jul 31 13:02:37.102948 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.102922    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"]
Jul 31 13:02:37.106755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.106737    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:37.112199 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.112177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:37.112407 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.112386    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558ec461-aa59-4217-a669-309c134b7664-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294\" (UID: \"558ec461-aa59-4217-a669-309c134b7664\") " pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:37.112551 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.112535    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5r4\" (UniqueName: \"kubernetes.io/projected/558ec461-aa59-4217-a669-309c134b7664-kube-api-access-5z5r4\") pod \"payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294\" (UID: \"558ec461-aa59-4217-a669-309c134b7664\") " pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:37.112842 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:37.112823    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-serving-cert: secret "maas-api-e2e-shared-a-96b3e0-serving-cert" not found
Jul 31 13:02:37.114807 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:02:37.113604    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls podName:01fd4400-bb38-4f3a-9eb7-4119889c64fa nodeName:}" failed. No retries permitted until 2026-07-31 13:02:38.112974226 +0000 UTC m=+2846.100195762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls") pod "maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2" (UID: "01fd4400-bb38-4f3a-9eb7-4119889c64fa") : secret "maas-api-e2e-shared-a-96b3e0-serving-cert" not found
Jul 31 13:02:37.119431 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.119311    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"]
Jul 31 13:02:37.213829 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.213793    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d1041-1a96-407f-a1ff-dac0e9a294bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q\" (UID: \"931d1041-1a96-407f-a1ff-dac0e9a294bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:37.214009 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.213874    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558ec461-aa59-4217-a669-309c134b7664-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294\" (UID: \"558ec461-aa59-4217-a669-309c134b7664\") " pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:37.214009 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.213899    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gdgg\" (UniqueName: \"kubernetes.io/projected/931d1041-1a96-407f-a1ff-dac0e9a294bf-kube-api-access-9gdgg\") pod \"payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q\" (UID: \"931d1041-1a96-407f-a1ff-dac0e9a294bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:37.214009 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.213914    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5r4\" (UniqueName: \"kubernetes.io/projected/558ec461-aa59-4217-a669-309c134b7664-kube-api-access-5z5r4\") pod \"payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294\" (UID: \"558ec461-aa59-4217-a669-309c134b7664\") " pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:37.214638 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.214612    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558ec461-aa59-4217-a669-309c134b7664-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294\" (UID: \"558ec461-aa59-4217-a669-309c134b7664\") " pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:37.222381 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.222353    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5r4\" (UniqueName: \"kubernetes.io/projected/558ec461-aa59-4217-a669-309c134b7664-kube-api-access-5z5r4\") pod \"payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294\" (UID: \"558ec461-aa59-4217-a669-309c134b7664\") " pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:37.315373 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.315339    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d1041-1a96-407f-a1ff-dac0e9a294bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q\" (UID: \"931d1041-1a96-407f-a1ff-dac0e9a294bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:37.315554 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.315489    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gdgg\" (UniqueName: \"kubernetes.io/projected/931d1041-1a96-407f-a1ff-dac0e9a294bf-kube-api-access-9gdgg\") pod \"payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q\" (UID: \"931d1041-1a96-407f-a1ff-dac0e9a294bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:37.316037 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.316014    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d1041-1a96-407f-a1ff-dac0e9a294bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q\" (UID: \"931d1041-1a96-407f-a1ff-dac0e9a294bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:37.325596 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.325533    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gdgg\" (UniqueName: \"kubernetes.io/projected/931d1041-1a96-407f-a1ff-dac0e9a294bf-kube-api-access-9gdgg\") pod \"payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q\" (UID: \"931d1041-1a96-407f-a1ff-dac0e9a294bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:37.384132 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.384093    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:37.419212 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.419180    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:37.532367 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.532343    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"]
Jul 31 13:02:37.534861 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:02:37.534826    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558ec461_aa59_4217_a669_309c134b7664.slice/crio-e4b868f062ea7d697f761f6299165e70d620d16c8ce1a40193e0cc97aa866a32 WatchSource:0}: Error finding container e4b868f062ea7d697f761f6299165e70d620d16c8ce1a40193e0cc97aa866a32: Status 404 returned error can't find the container with id e4b868f062ea7d697f761f6299165e70d620d16c8ce1a40193e0cc97aa866a32
Jul 31 13:02:37.570878 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:37.570853    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"]
Jul 31 13:02:37.572537 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:02:37.572514    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931d1041_1a96_407f_a1ff_dac0e9a294bf.slice/crio-a7d1d703c03540891ccc48c538caffcc07e0676a1de3d2ac3a57fe3d4f532204 WatchSource:0}: Error finding container a7d1d703c03540891ccc48c538caffcc07e0676a1de3d2ac3a57fe3d4f532204: Status 404 returned error can't find the container with id a7d1d703c03540891ccc48c538caffcc07e0676a1de3d2ac3a57fe3d4f532204
Jul 31 13:02:38.126816 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.126780    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:38.129179 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.129158    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:38.216029 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.215995    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:38.355927 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.355903    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"]
Jul 31 13:02:38.357581 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:02:38.357554    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fd4400_bb38_4f3a_9eb7_4119889c64fa.slice/crio-e084afdfefa965bf996642f230d42ef40fe792747079239d4d3a6c7dea4b1bb5 WatchSource:0}: Error finding container e084afdfefa965bf996642f230d42ef40fe792747079239d4d3a6c7dea4b1bb5: Status 404 returned error can't find the container with id e084afdfefa965bf996642f230d42ef40fe792747079239d4d3a6c7dea4b1bb5
Jul 31 13:02:38.405784 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.405751    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q" event={"ID":"931d1041-1a96-407f-a1ff-dac0e9a294bf","Type":"ContainerStarted","Data":"5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f"}
Jul 31 13:02:38.405944 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.405809    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q" event={"ID":"931d1041-1a96-407f-a1ff-dac0e9a294bf","Type":"ContainerStarted","Data":"a7d1d703c03540891ccc48c538caffcc07e0676a1de3d2ac3a57fe3d4f532204"}
Jul 31 13:02:38.405944 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.405917    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:38.406884 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.406860    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2" event={"ID":"01fd4400-bb38-4f3a-9eb7-4119889c64fa","Type":"ContainerStarted","Data":"e084afdfefa965bf996642f230d42ef40fe792747079239d4d3a6c7dea4b1bb5"}
Jul 31 13:02:38.408600 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.408561    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294" event={"ID":"558ec461-aa59-4217-a669-309c134b7664","Type":"ContainerStarted","Data":"3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6"}
Jul 31 13:02:38.408717 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.408614    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294" event={"ID":"558ec461-aa59-4217-a669-309c134b7664","Type":"ContainerStarted","Data":"e4b868f062ea7d697f761f6299165e70d620d16c8ce1a40193e0cc97aa866a32"}
Jul 31 13:02:38.408717 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.408699    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:38.423907 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.423866    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q" podStartSLOduration=1.4238533979999999 podStartE2EDuration="1.423853398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:02:38.423004228 +0000 UTC m=+2846.410225768" watchObservedRunningTime="2026-07-31 13:02:38.423853398 +0000 UTC m=+2846.411074937"
Jul 31 13:02:38.440060 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:38.440008    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294" podStartSLOduration=1.439995967 podStartE2EDuration="1.439995967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:02:38.438439228 +0000 UTC m=+2846.425660768" watchObservedRunningTime="2026-07-31 13:02:38.439995967 +0000 UTC m=+2846.427217485"
Jul 31 13:02:39.415438 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:39.415390    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2" event={"ID":"01fd4400-bb38-4f3a-9eb7-4119889c64fa","Type":"ContainerStarted","Data":"a7a5ee46df57d4846566ec2d35f0188cc3a60d593bc9170edd131f8b242037bc"}
Jul 31 13:02:39.415781 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:39.415763    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:39.433506 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:39.433449    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2" podStartSLOduration=2.962522942 podStartE2EDuration="3.433435546s" totalImagesPullingTime="470.912604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:02:39.43187586 +0000 UTC m=+2847.419097418" watchObservedRunningTime="2026-07-31 13:02:39.433435546 +0000 UTC m=+2847.420657085"
Jul 31 13:02:45.426403 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:45.426374    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:02:49.385064 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.385028    2586 patch_prober.go:28] interesting pod/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 13:02:49.385518 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.385086    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 13:02:49.417156 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.417130    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:02:49.417334 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.417212    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:02:49.477943 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.477907    2586 generic.go:356] "Generic (PLEG): container finished" podID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerID="bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5" exitCode=0
Jul 31 13:02:49.478093 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.477971    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" event={"ID":"ba4af621-bf34-49a3-8870-ecbd873d50b2","Type":"ContainerDied","Data":"bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5"}
Jul 31 13:02:49.478408 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.478392    2586 scope.go:117] "RemoveContainer" containerID="bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5"
Jul 31 13:02:49.479449 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.479425    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:02:49.479524 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.479488    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:02:49.479524 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:49.479519    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 13:02:50.485842 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:50.485806    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" event={"ID":"ba4af621-bf34-49a3-8870-ecbd873d50b2","Type":"ContainerStarted","Data":"b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38"}
Jul 31 13:02:50.782510 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:50.782421    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:50.782693 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:50.782672    2586 patch_prober.go:28] interesting pod/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 13:02:50.782752 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:50.782730    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 13:02:51.783090 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:51.783057    2586 patch_prober.go:28] interesting pod/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 13:02:51.783516 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:51.783114    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 13:02:52.783445 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:52.783403    2586 patch_prober.go:28] interesting pod/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 13:02:52.783875 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:52.783469    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 13:02:53.782570 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:53.782531    2586 patch_prober.go:28] interesting pod/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 13:02:53.782792 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:53.782598    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 13:02:54.782948 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:54.782914    2586 patch_prober.go:28] interesting pod/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 13:02:54.783353 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:54.782970    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 13:02:55.786400 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:55.786370    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:55.786783 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:55.786623    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:02:55.787208 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:02:55.787193    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:03:06.907235 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:06.907153    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"]
Jul 31 13:03:06.910745 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:06.910729    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:06.913146 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:06.913121    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e55b67-serving-cert\""
Jul 31 13:03:06.922117 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:06.922095    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"]
Jul 31 13:03:07.005899 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.005868    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.006065 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.005924    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nst2n\" (UniqueName: \"kubernetes.io/projected/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-kube-api-access-nst2n\") pod \"maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.006065 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.006004    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.106663 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.106623    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.106859 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.106712    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nst2n\" (UniqueName: \"kubernetes.io/projected/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-kube-api-access-nst2n\") pod \"maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.106859 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.106750    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.107418 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.107393    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.109012 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.108993    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.115145 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.115123    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nst2n\" (UniqueName: \"kubernetes.io/projected/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-kube-api-access-nst2n\") pod \"maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.225065 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.224977    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:07.363737 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.363711    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"]
Jul 31 13:03:07.365185 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:03:07.365153    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7d0d5d_d6d4_4190_af32_33163ce28fc4.slice/crio-5d7ad36ccdd9a17a5df6a69aaea652857751a7076ae8ccb47a2f06c07213f6fd WatchSource:0}: Error finding container 5d7ad36ccdd9a17a5df6a69aaea652857751a7076ae8ccb47a2f06c07213f6fd: Status 404 returned error can't find the container with id 5d7ad36ccdd9a17a5df6a69aaea652857751a7076ae8ccb47a2f06c07213f6fd
Jul 31 13:03:07.467219 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.467193    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"]
Jul 31 13:03:07.472138 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.472116    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:03:07.474643 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.474619    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-e55b67-dockercfg-49l45\""
Jul 31 13:03:07.474779 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.474619    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-e55b67\""
Jul 31 13:03:07.480962 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.480940    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"]
Jul 31 13:03:07.511304 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.511280    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2895\" (UniqueName: \"kubernetes.io/projected/29b92a90-2104-4ba9-b336-54fb93b3f16a-kube-api-access-b2895\") pod \"payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j\" (UID: \"29b92a90-2104-4ba9-b336-54fb93b3f16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:03:07.511407 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.511348    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b92a90-2104-4ba9-b336-54fb93b3f16a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j\" (UID: \"29b92a90-2104-4ba9-b336-54fb93b3f16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:03:07.567731 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.567700    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml" event={"ID":"ae7d0d5d-d6d4-4190-af32-33163ce28fc4","Type":"ContainerStarted","Data":"5d7ad36ccdd9a17a5df6a69aaea652857751a7076ae8ccb47a2f06c07213f6fd"}
Jul 31 13:03:07.612305 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.612249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2895\" (UniqueName: \"kubernetes.io/projected/29b92a90-2104-4ba9-b336-54fb93b3f16a-kube-api-access-b2895\") pod \"payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j\" (UID: \"29b92a90-2104-4ba9-b336-54fb93b3f16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:03:07.612502 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.612381    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b92a90-2104-4ba9-b336-54fb93b3f16a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j\" (UID: \"29b92a90-2104-4ba9-b336-54fb93b3f16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:03:07.613168 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.613144    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b92a90-2104-4ba9-b336-54fb93b3f16a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j\" (UID: \"29b92a90-2104-4ba9-b336-54fb93b3f16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:03:07.621447 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.621424    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2895\" (UniqueName: \"kubernetes.io/projected/29b92a90-2104-4ba9-b336-54fb93b3f16a-kube-api-access-b2895\") pod \"payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j\" (UID: \"29b92a90-2104-4ba9-b336-54fb93b3f16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:03:07.784453 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.784429    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:03:07.915571 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:07.915546    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"]
Jul 31 13:03:07.918281 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:03:07.918238    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b92a90_2104_4ba9_b336_54fb93b3f16a.slice/crio-ed1cd38b5ba29e21972c447fd09820919e76ea17919ee4bf5d02cedebfeb5d3d WatchSource:0}: Error finding container ed1cd38b5ba29e21972c447fd09820919e76ea17919ee4bf5d02cedebfeb5d3d: Status 404 returned error can't find the container with id ed1cd38b5ba29e21972c447fd09820919e76ea17919ee4bf5d02cedebfeb5d3d
Jul 31 13:03:08.574133 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:08.574098    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml" event={"ID":"ae7d0d5d-d6d4-4190-af32-33163ce28fc4","Type":"ContainerStarted","Data":"1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac"}
Jul 31 13:03:08.574348 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:08.574294    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:08.575775 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:08.575749    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j" event={"ID":"29b92a90-2104-4ba9-b336-54fb93b3f16a","Type":"ContainerStarted","Data":"c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d"}
Jul 31 13:03:08.575882 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:08.575794    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j" event={"ID":"29b92a90-2104-4ba9-b336-54fb93b3f16a","Type":"ContainerStarted","Data":"ed1cd38b5ba29e21972c447fd09820919e76ea17919ee4bf5d02cedebfeb5d3d"}
Jul 31 13:03:08.575882 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:08.575825    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:03:08.590664 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:08.590622    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml" podStartSLOduration=2.172862244 podStartE2EDuration="2.590609133s" totalImagesPullingTime="417.746889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:03:08.588992843 +0000 UTC m=+2876.576214383" watchObservedRunningTime="2026-07-31 13:03:08.590609133 +0000 UTC m=+2876.577830672"
Jul 31 13:03:08.603919 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:08.603866    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j" podStartSLOduration=1.603848607 podStartE2EDuration="1.603848607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:03:08.603648941 +0000 UTC m=+2876.590870483" watchObservedRunningTime="2026-07-31 13:03:08.603848607 +0000 UTC m=+2876.591070148"
Jul 31 13:03:14.586600 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:14.586572    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:03:19.583234 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:03:19.583204    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:06:12.861983 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:06:12.861902    2586 scope.go:117] "RemoveContainer" containerID="29d328a26a7e2206df4f97604b2c763857488e27202185640d4a6648c6587b70"
Jul 31 13:07:57.369820 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.369785    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"]
Jul 31 13:07:57.372934 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.372919    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.375873 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.375848    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"kube-root-ca.crt\""
Jul 31 13:07:57.376983 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.376963    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"default-dockercfg-hj927\""
Jul 31 13:07:57.377111 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.376984    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"sub-test-model-e55b67-kserve-self-signed-certs\""
Jul 31 13:07:57.377111 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.376984    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"openshift-service-ca.crt\""
Jul 31 13:07:57.381307 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.381286    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"]
Jul 31 13:07:57.469255 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.469217    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959a17cf-2811-4388-b0dc-4b2d13875fb9-tls-certs\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.469458 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.469262    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-model-cache\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.469458 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.469311    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-home\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.469458 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.469359    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-dshm\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.469614 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.469450    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-tmp-dir\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.469614 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.469478    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcz9f\" (UniqueName: \"kubernetes.io/projected/959a17cf-2811-4388-b0dc-4b2d13875fb9-kube-api-access-gcz9f\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.570066 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.570034    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959a17cf-2811-4388-b0dc-4b2d13875fb9-tls-certs\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.570066 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.570070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-model-cache\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.570316 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.570092    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-home\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.570316 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.570245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-dshm\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.570435 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.570323    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-tmp-dir\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.570435 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.570349    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcz9f\" (UniqueName: \"kubernetes.io/projected/959a17cf-2811-4388-b0dc-4b2d13875fb9-kube-api-access-gcz9f\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.570550 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.570480    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-model-cache\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.570619 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.570591    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-home\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.570709 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.570688    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-tmp-dir\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.572478 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.572450    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-dshm\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.572774 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.572753    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959a17cf-2811-4388-b0dc-4b2d13875fb9-tls-certs\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.579775 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.579742    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcz9f\" (UniqueName: \"kubernetes.io/projected/959a17cf-2811-4388-b0dc-4b2d13875fb9-kube-api-access-gcz9f\") pod \"sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") " pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.682825 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.682789    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:07:57.814498 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:57.814475    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"]
Jul 31 13:07:57.817439 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:07:57.817408    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod959a17cf_2811_4388_b0dc_4b2d13875fb9.slice/crio-8551d27e0ff447af266ca8c16aa275cc824e5f5969d5648175c337a67890ac2f WatchSource:0}: Error finding container 8551d27e0ff447af266ca8c16aa275cc824e5f5969d5648175c337a67890ac2f: Status 404 returned error can't find the container with id 8551d27e0ff447af266ca8c16aa275cc824e5f5969d5648175c337a67890ac2f
Jul 31 13:07:58.049547 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:58.049460    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx" event={"ID":"959a17cf-2811-4388-b0dc-4b2d13875fb9","Type":"ContainerStarted","Data":"dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c"}
Jul 31 13:07:58.049547 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:58.049502    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx" event={"ID":"959a17cf-2811-4388-b0dc-4b2d13875fb9","Type":"ContainerStarted","Data":"8551d27e0ff447af266ca8c16aa275cc824e5f5969d5648175c337a67890ac2f"}
Jul 31 13:07:58.066598 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:07:58.066554    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx" podStartSLOduration=1.066537893 podStartE2EDuration="1.066537893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:07:58.065077599 +0000 UTC m=+3166.052299139" watchObservedRunningTime="2026-07-31 13:07:58.066537893 +0000 UTC m=+3166.053759433"
Jul 31 13:08:07.683587 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:08:07.683546    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:08:07.683587 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:08:07.683599    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:08:07.696123 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:08:07.696097    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:08:08.110634 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:08:08.110557    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:09:09.599750 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:09.599670    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"]
Jul 31 13:09:24.618544 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:24.618502    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx" podUID="959a17cf-2811-4388-b0dc-4b2d13875fb9" containerName="main" containerID="cri-o://dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c" gracePeriod=45
Jul 31 13:09:24.875819 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:24.875759    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:09:25.003404 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.003372    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959a17cf-2811-4388-b0dc-4b2d13875fb9-tls-certs\") pod \"959a17cf-2811-4388-b0dc-4b2d13875fb9\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") "
Jul 31 13:09:25.003404 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.003413    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-model-cache\") pod \"959a17cf-2811-4388-b0dc-4b2d13875fb9\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") "
Jul 31 13:09:25.003619 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.003459    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-tmp-dir\") pod \"959a17cf-2811-4388-b0dc-4b2d13875fb9\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") "
Jul 31 13:09:25.003619 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.003548    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-home\") pod \"959a17cf-2811-4388-b0dc-4b2d13875fb9\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") "
Jul 31 13:09:25.003619 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.003610    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-dshm\") pod \"959a17cf-2811-4388-b0dc-4b2d13875fb9\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") "
Jul 31 13:09:25.003749 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.003638    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcz9f\" (UniqueName: \"kubernetes.io/projected/959a17cf-2811-4388-b0dc-4b2d13875fb9-kube-api-access-gcz9f\") pod \"959a17cf-2811-4388-b0dc-4b2d13875fb9\" (UID: \"959a17cf-2811-4388-b0dc-4b2d13875fb9\") "
Jul 31 13:09:25.003806 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.003752    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "959a17cf-2811-4388-b0dc-4b2d13875fb9" (UID: "959a17cf-2811-4388-b0dc-4b2d13875fb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:09:25.003806 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.003769    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "959a17cf-2811-4388-b0dc-4b2d13875fb9" (UID: "959a17cf-2811-4388-b0dc-4b2d13875fb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:09:25.003919 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.003888    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-home" (OuterVolumeSpecName: "home") pod "959a17cf-2811-4388-b0dc-4b2d13875fb9" (UID: "959a17cf-2811-4388-b0dc-4b2d13875fb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:09:25.004026 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.004005    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:09:25.004099 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.004032    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:09:25.004099 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.004044    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:09:25.005656 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.005628    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959a17cf-2811-4388-b0dc-4b2d13875fb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "959a17cf-2811-4388-b0dc-4b2d13875fb9" (UID: "959a17cf-2811-4388-b0dc-4b2d13875fb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 13:09:25.005768 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.005726    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959a17cf-2811-4388-b0dc-4b2d13875fb9-kube-api-access-gcz9f" (OuterVolumeSpecName: "kube-api-access-gcz9f") pod "959a17cf-2811-4388-b0dc-4b2d13875fb9" (UID: "959a17cf-2811-4388-b0dc-4b2d13875fb9"). InnerVolumeSpecName "kube-api-access-gcz9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:09:25.005845 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.005824    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-dshm" (OuterVolumeSpecName: "dshm") pod "959a17cf-2811-4388-b0dc-4b2d13875fb9" (UID: "959a17cf-2811-4388-b0dc-4b2d13875fb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:09:25.105338 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.105313    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959a17cf-2811-4388-b0dc-4b2d13875fb9-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:09:25.105338 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.105335    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcz9f\" (UniqueName: \"kubernetes.io/projected/959a17cf-2811-4388-b0dc-4b2d13875fb9-kube-api-access-gcz9f\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:09:25.105522 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.105347    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959a17cf-2811-4388-b0dc-4b2d13875fb9-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:09:25.499841 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.499811    2586 generic.go:356] "Generic (PLEG): container finished" podID="959a17cf-2811-4388-b0dc-4b2d13875fb9" containerID="dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c" exitCode=0
Jul 31 13:09:25.500010 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.499851    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx" event={"ID":"959a17cf-2811-4388-b0dc-4b2d13875fb9","Type":"ContainerDied","Data":"dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c"}
Jul 31 13:09:25.500010 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.499868    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"
Jul 31 13:09:25.500010 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.499878    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx" event={"ID":"959a17cf-2811-4388-b0dc-4b2d13875fb9","Type":"ContainerDied","Data":"8551d27e0ff447af266ca8c16aa275cc824e5f5969d5648175c337a67890ac2f"}
Jul 31 13:09:25.500010 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.499899    2586 scope.go:117] "RemoveContainer" containerID="dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c"
Jul 31 13:09:25.513106 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.513087    2586 scope.go:117] "RemoveContainer" containerID="dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c"
Jul 31 13:09:25.513411 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:09:25.513388    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c\": container with ID starting with dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c not found: ID does not exist" containerID="dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c"
Jul 31 13:09:25.513493 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.513423    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c"} err="failed to get container status \"dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c\": rpc error: code = NotFound desc = could not find container \"dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c\": container with ID starting with dd44bd1653f375d3e8300d35aaa6679c3c9eb1c9b83e13c34ddbe958ccbb130c not found: ID does not exist"
Jul 31 13:09:25.526168 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.526143    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"]
Jul 31 13:09:25.529549 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:25.529529    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/sub-test-model-e55b67-kserve-85b7cf47ff-cv5dx"]
Jul 31 13:09:26.485177 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:09:26.485142    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="959a17cf-2811-4388-b0dc-4b2d13875fb9" path="/var/lib/kubelet/pods/959a17cf-2811-4388-b0dc-4b2d13875fb9/volumes"
Jul 31 13:10:12.242857 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.242819    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"]
Jul 31 13:10:12.243455 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.243445    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959a17cf-2811-4388-b0dc-4b2d13875fb9" containerName="main"
Jul 31 13:10:12.243520 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.243460    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="959a17cf-2811-4388-b0dc-4b2d13875fb9" containerName="main"
Jul 31 13:10:12.243559 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.243522    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="959a17cf-2811-4388-b0dc-4b2d13875fb9" containerName="main"
Jul 31 13:10:12.246143 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.246127    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.248678 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.248651    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"openshift-service-ca.crt\""
Jul 31 13:10:12.249814 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.249677    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"default-dockercfg-hj927\""
Jul 31 13:10:12.249814 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.249703    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"kube-root-ca.crt\""
Jul 31 13:10:12.249814 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.249718    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"rate-test-model-e55b67-kserve-self-signed-certs\""
Jul 31 13:10:12.257309 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.257254    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"]
Jul 31 13:10:12.353681 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.353641    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkn7\" (UniqueName: \"kubernetes.io/projected/386ed976-cc12-4c44-b696-c3b8892c01d5-kube-api-access-6rkn7\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.353681 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.353679    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-dshm\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.353911 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.353704    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-model-cache\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.353911 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.353725    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-tmp-dir\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.353911 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.353757    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-home\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.353911 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.353880    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ed976-cc12-4c44-b696-c3b8892c01d5-tls-certs\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.454928 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.454891    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkn7\" (UniqueName: \"kubernetes.io/projected/386ed976-cc12-4c44-b696-c3b8892c01d5-kube-api-access-6rkn7\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.454928 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.454929    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-dshm\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.455196 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.454956    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-model-cache\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.455196 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.455071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-tmp-dir\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.455196 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.455126    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-home\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.455402 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.455230    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ed976-cc12-4c44-b696-c3b8892c01d5-tls-certs\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.455402 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.455366    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-model-cache\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.455483 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.455450    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-tmp-dir\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.455521 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.455483    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-home\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.457299 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.457255    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-dshm\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.457447 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.457429    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"rate-test-model-e55b67-kserve-self-signed-certs\""
Jul 31 13:10:12.462258 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.462241    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"kube-root-ca.crt\""
Jul 31 13:10:12.468181 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.468160    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ed976-cc12-4c44-b696-c3b8892c01d5-tls-certs\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.472980 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.472961    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"openshift-service-ca.crt\""
Jul 31 13:10:12.483417 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.483391    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkn7\" (UniqueName: \"kubernetes.io/projected/386ed976-cc12-4c44-b696-c3b8892c01d5-kube-api-access-6rkn7\") pod \"rate-test-model-e55b67-kserve-764d6b8c95-wbdj5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") " pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.560426 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.560350    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e55b67\"/\"default-dockercfg-hj927\""
Jul 31 13:10:12.569209 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.569184    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:12.722565 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.722512    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"]
Jul 31 13:10:12.724856 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:10:12.724829    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod386ed976_cc12_4c44_b696_c3b8892c01d5.slice/crio-6359b586d9fc1acc6b42b40ee8c553bc9ed14b8b2a69f4483167eb6d705afc5d WatchSource:0}: Error finding container 6359b586d9fc1acc6b42b40ee8c553bc9ed14b8b2a69f4483167eb6d705afc5d: Status 404 returned error can't find the container with id 6359b586d9fc1acc6b42b40ee8c553bc9ed14b8b2a69f4483167eb6d705afc5d
Jul 31 13:10:12.735952 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:12.735928    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5" event={"ID":"386ed976-cc12-4c44-b696-c3b8892c01d5","Type":"ContainerStarted","Data":"6359b586d9fc1acc6b42b40ee8c553bc9ed14b8b2a69f4483167eb6d705afc5d"}
Jul 31 13:10:13.743371 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:13.743333    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5" event={"ID":"386ed976-cc12-4c44-b696-c3b8892c01d5","Type":"ContainerStarted","Data":"9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396"}
Jul 31 13:10:13.764766 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:13.764705    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5" podStartSLOduration=1.7646864180000001 podStartE2EDuration="1.764686418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:10:13.764018823 +0000 UTC m=+3301.751240364" watchObservedRunningTime="2026-07-31 13:10:13.764686418 +0000 UTC m=+3301.751907963"
Jul 31 13:10:22.570157 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:22.570120    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:22.570687 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:22.570188    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:22.583227 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:22.583201    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:10:22.801663 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:10:22.801631    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:11:49.405080 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:11:49.405042    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"]
Jul 31 13:12:04.425941 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.425854    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5" podUID="386ed976-cc12-4c44-b696-c3b8892c01d5" containerName="main" containerID="cri-o://9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396" gracePeriod=45
Jul 31 13:12:04.680232 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.680172    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:12:04.708498 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.708468    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-dshm\") pod \"386ed976-cc12-4c44-b696-c3b8892c01d5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") "
Jul 31 13:12:04.708683 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.708518    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-home\") pod \"386ed976-cc12-4c44-b696-c3b8892c01d5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") "
Jul 31 13:12:04.708683 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.708577    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-tmp-dir\") pod \"386ed976-cc12-4c44-b696-c3b8892c01d5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") "
Jul 31 13:12:04.708683 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.708608    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ed976-cc12-4c44-b696-c3b8892c01d5-tls-certs\") pod \"386ed976-cc12-4c44-b696-c3b8892c01d5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") "
Jul 31 13:12:04.708683 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.708671    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-model-cache\") pod \"386ed976-cc12-4c44-b696-c3b8892c01d5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") "
Jul 31 13:12:04.708916 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.708703    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rkn7\" (UniqueName: \"kubernetes.io/projected/386ed976-cc12-4c44-b696-c3b8892c01d5-kube-api-access-6rkn7\") pod \"386ed976-cc12-4c44-b696-c3b8892c01d5\" (UID: \"386ed976-cc12-4c44-b696-c3b8892c01d5\") "
Jul 31 13:12:04.709333 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.709289    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-home" (OuterVolumeSpecName: "home") pod "386ed976-cc12-4c44-b696-c3b8892c01d5" (UID: "386ed976-cc12-4c44-b696-c3b8892c01d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:12:04.709333 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.709298    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "386ed976-cc12-4c44-b696-c3b8892c01d5" (UID: "386ed976-cc12-4c44-b696-c3b8892c01d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:12:04.709519 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.709430    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "386ed976-cc12-4c44-b696-c3b8892c01d5" (UID: "386ed976-cc12-4c44-b696-c3b8892c01d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:12:04.711241 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.711187    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-dshm" (OuterVolumeSpecName: "dshm") pod "386ed976-cc12-4c44-b696-c3b8892c01d5" (UID: "386ed976-cc12-4c44-b696-c3b8892c01d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:12:04.711591 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.711555    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386ed976-cc12-4c44-b696-c3b8892c01d5-kube-api-access-6rkn7" (OuterVolumeSpecName: "kube-api-access-6rkn7") pod "386ed976-cc12-4c44-b696-c3b8892c01d5" (UID: "386ed976-cc12-4c44-b696-c3b8892c01d5"). InnerVolumeSpecName "kube-api-access-6rkn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:12:04.711706 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.711638    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386ed976-cc12-4c44-b696-c3b8892c01d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "386ed976-cc12-4c44-b696-c3b8892c01d5" (UID: "386ed976-cc12-4c44-b696-c3b8892c01d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 13:12:04.810136 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.810110    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:12:04.810136 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.810132    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ed976-cc12-4c44-b696-c3b8892c01d5-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:12:04.810316 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.810142    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:12:04.810316 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.810153    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rkn7\" (UniqueName: \"kubernetes.io/projected/386ed976-cc12-4c44-b696-c3b8892c01d5-kube-api-access-6rkn7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:12:04.810316 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.810162    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:12:04.810316 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:04.810170    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ed976-cc12-4c44-b696-c3b8892c01d5-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:12:05.308954 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:05.308913    2586 generic.go:356] "Generic (PLEG): container finished" podID="386ed976-cc12-4c44-b696-c3b8892c01d5" containerID="9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396" exitCode=0
Jul 31 13:12:05.309138 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:05.308959    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"
Jul 31 13:12:05.309138 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:05.308996    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5" event={"ID":"386ed976-cc12-4c44-b696-c3b8892c01d5","Type":"ContainerDied","Data":"9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396"}
Jul 31 13:12:05.309138 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:05.309038    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5" event={"ID":"386ed976-cc12-4c44-b696-c3b8892c01d5","Type":"ContainerDied","Data":"6359b586d9fc1acc6b42b40ee8c553bc9ed14b8b2a69f4483167eb6d705afc5d"}
Jul 31 13:12:05.309138 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:05.309057    2586 scope.go:117] "RemoveContainer" containerID="9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396"
Jul 31 13:12:05.318517 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:05.318496    2586 scope.go:117] "RemoveContainer" containerID="9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396"
Jul 31 13:12:05.318833 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:12:05.318807    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396\": container with ID starting with 9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396 not found: ID does not exist" containerID="9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396"
Jul 31 13:12:05.318921 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:05.318848    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396"} err="failed to get container status \"9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396\": rpc error: code = NotFound desc = could not find container \"9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396\": container with ID starting with 9392596980d4d2300b2d2c9f4ef03020b4075b0a5c2dd565cfd3ea522ba76396 not found: ID does not exist"
Jul 31 13:12:05.331646 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:05.331614    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"]
Jul 31 13:12:05.335062 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:05.335034    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e55b67/rate-test-model-e55b67-kserve-764d6b8c95-wbdj5"]
Jul 31 13:12:06.485434 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:06.485402    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386ed976-cc12-4c44-b696-c3b8892c01d5" path="/var/lib/kubelet/pods/386ed976-cc12-4c44-b696-c3b8892c01d5/volumes"
Jul 31 13:12:15.355015 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.354973    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"]
Jul 31 13:12:15.355595 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.355576    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386ed976-cc12-4c44-b696-c3b8892c01d5" containerName="main"
Jul 31 13:12:15.355595 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.355597    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="386ed976-cc12-4c44-b696-c3b8892c01d5" containerName="main"
Jul 31 13:12:15.355747 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.355734    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="386ed976-cc12-4c44-b696-c3b8892c01d5" containerName="main"
Jul 31 13:12:15.360130 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.360110    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:15.362605 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.362587    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fd6ea7\""
Jul 31 13:12:15.362724 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.362608    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fd6ea7-dockercfg-784ss\""
Jul 31 13:12:15.370693 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.370633    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"]
Jul 31 13:12:15.373110 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.373091    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"]
Jul 31 13:12:15.376770 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.376753    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:15.385792 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.385773    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"]
Jul 31 13:12:15.409789 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.409753    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26497c12-1a75-42a9-8a48-c8fc230c25e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf\" (UID: \"26497c12-1a75-42a9-8a48-c8fc230c25e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:15.409913 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.409879    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkd89\" (UniqueName: \"kubernetes.io/projected/26497c12-1a75-42a9-8a48-c8fc230c25e4-kube-api-access-qkd89\") pod \"payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf\" (UID: \"26497c12-1a75-42a9-8a48-c8fc230c25e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:15.409955 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.409931    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5\" (UID: \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:15.410013 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.409997    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwr9z\" (UniqueName: \"kubernetes.io/projected/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-kube-api-access-hwr9z\") pod \"payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5\" (UID: \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:15.510833 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.510794    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwr9z\" (UniqueName: \"kubernetes.io/projected/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-kube-api-access-hwr9z\") pod \"payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5\" (UID: \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:15.511003 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.510851    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26497c12-1a75-42a9-8a48-c8fc230c25e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf\" (UID: \"26497c12-1a75-42a9-8a48-c8fc230c25e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:15.511003 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.510903    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkd89\" (UniqueName: \"kubernetes.io/projected/26497c12-1a75-42a9-8a48-c8fc230c25e4-kube-api-access-qkd89\") pod \"payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf\" (UID: \"26497c12-1a75-42a9-8a48-c8fc230c25e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:15.511003 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.510918    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5\" (UID: \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:15.511687 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.511667    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26497c12-1a75-42a9-8a48-c8fc230c25e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf\" (UID: \"26497c12-1a75-42a9-8a48-c8fc230c25e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:15.511738 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.511678    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5\" (UID: \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:15.519672 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.519646    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwr9z\" (UniqueName: \"kubernetes.io/projected/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-kube-api-access-hwr9z\") pod \"payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5\" (UID: \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:15.519842 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.519822    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkd89\" (UniqueName: \"kubernetes.io/projected/26497c12-1a75-42a9-8a48-c8fc230c25e4-kube-api-access-qkd89\") pod \"payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf\" (UID: \"26497c12-1a75-42a9-8a48-c8fc230c25e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:15.672291 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.672239    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:15.688688 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.688665    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:15.817505 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:15.817476    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"]
Jul 31 13:12:15.819034 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:12:15.819001    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead09e73_2bf2_4bc7_93c9_d28eae8ca5e7.slice/crio-bd9fe262fd702346c7795566152fb8d7e204b9d5d613652d5e497d6d08b01d99 WatchSource:0}: Error finding container bd9fe262fd702346c7795566152fb8d7e204b9d5d613652d5e497d6d08b01d99: Status 404 returned error can't find the container with id bd9fe262fd702346c7795566152fb8d7e204b9d5d613652d5e497d6d08b01d99
Jul 31 13:12:16.054136 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:16.054107    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"]
Jul 31 13:12:16.055858 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:12:16.055832    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26497c12_1a75_42a9_8a48_c8fc230c25e4.slice/crio-22c27bfec2c8fe946a7cf19d3bec4e3e5f0368c4f8ed5afd6034526d9e1c34a6 WatchSource:0}: Error finding container 22c27bfec2c8fe946a7cf19d3bec4e3e5f0368c4f8ed5afd6034526d9e1c34a6: Status 404 returned error can't find the container with id 22c27bfec2c8fe946a7cf19d3bec4e3e5f0368c4f8ed5afd6034526d9e1c34a6
Jul 31 13:12:16.365524 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:16.365434    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf" event={"ID":"26497c12-1a75-42a9-8a48-c8fc230c25e4","Type":"ContainerStarted","Data":"4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451"}
Jul 31 13:12:16.365524 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:16.365484    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf" event={"ID":"26497c12-1a75-42a9-8a48-c8fc230c25e4","Type":"ContainerStarted","Data":"22c27bfec2c8fe946a7cf19d3bec4e3e5f0368c4f8ed5afd6034526d9e1c34a6"}
Jul 31 13:12:16.366016 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:16.365614    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:16.367191 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:16.367163    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5" event={"ID":"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7","Type":"ContainerStarted","Data":"59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184"}
Jul 31 13:12:16.367315 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:16.367214    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5" event={"ID":"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7","Type":"ContainerStarted","Data":"bd9fe262fd702346c7795566152fb8d7e204b9d5d613652d5e497d6d08b01d99"}
Jul 31 13:12:16.367315 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:16.367242    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:16.391989 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:16.391947    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf" podStartSLOduration=1.391935431 podStartE2EDuration="1.391935431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:12:16.391347531 +0000 UTC m=+3424.378569071" watchObservedRunningTime="2026-07-31 13:12:16.391935431 +0000 UTC m=+3424.379156971"
Jul 31 13:12:16.407456 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:16.407413    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5" podStartSLOduration=1.407401737 podStartE2EDuration="1.407401737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:12:16.406942992 +0000 UTC m=+3424.394164533" watchObservedRunningTime="2026-07-31 13:12:16.407401737 +0000 UTC m=+3424.394623276"
Jul 31 13:12:27.374873 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:27.374842    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:12:27.375366 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:27.374913    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:12:38.450987 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.450954    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"]
Jul 31 13:12:38.454467 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.454450    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:38.456693 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.456671    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f59489-dockercfg-lpz68\""
Jul 31 13:12:38.456780 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.456671    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f59489\""
Jul 31 13:12:38.461587 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.461567    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"]
Jul 31 13:12:38.628302 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.628249    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzhm9\" (UniqueName: \"kubernetes.io/projected/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-kube-api-access-mzhm9\") pod \"payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5\" (UID: \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:38.628477 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.628319    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5\" (UID: \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:38.729806 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.729717    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5\" (UID: \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:38.729956 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.729827    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzhm9\" (UniqueName: \"kubernetes.io/projected/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-kube-api-access-mzhm9\") pod \"payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5\" (UID: \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:38.730428 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.730408    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5\" (UID: \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:38.738053 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.738028    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzhm9\" (UniqueName: \"kubernetes.io/projected/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-kube-api-access-mzhm9\") pod \"payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5\" (UID: \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:38.767892 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.767865    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:38.900241 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:38.900214    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"]
Jul 31 13:12:38.902516 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:12:38.902485    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92e1cf0_af4e_4d22_bfb8_57b2c01a51f2.slice/crio-e8f98218d4714b69906482bab373af3fb34e5d0cfa22d35a6cb7e6c18ce1e41c WatchSource:0}: Error finding container e8f98218d4714b69906482bab373af3fb34e5d0cfa22d35a6cb7e6c18ce1e41c: Status 404 returned error can't find the container with id e8f98218d4714b69906482bab373af3fb34e5d0cfa22d35a6cb7e6c18ce1e41c
Jul 31 13:12:39.488898 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:39.488857    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5" event={"ID":"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2","Type":"ContainerStarted","Data":"92e775be20a94096bfa59e91af8018efd6c1d95aa82634863f165c6ad74b0e69"}
Jul 31 13:12:39.489304 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:39.488956    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5" event={"ID":"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2","Type":"ContainerStarted","Data":"e8f98218d4714b69906482bab373af3fb34e5d0cfa22d35a6cb7e6c18ce1e41c"}
Jul 31 13:12:39.489304 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:39.488989    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:39.506010 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:39.505965    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5" podStartSLOduration=1.50595144 podStartE2EDuration="1.50595144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:12:39.505088907 +0000 UTC m=+3447.492310448" watchObservedRunningTime="2026-07-31 13:12:39.50595144 +0000 UTC m=+3447.493172979"
Jul 31 13:12:50.495861 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:50.495834    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:12:57.708595 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:57.708552    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pdrrk_d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2/payload-processing/0.log"
Jul 31 13:12:57.838332 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:12:57.838304    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5_b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2/payload-processing/0.log"
Jul 31 13:13:09.152496 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.152458    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"]
Jul 31 13:13:09.156116 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.156096    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.159655 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.159490    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-fd6ea7\"/\"openshift-service-ca.crt\""
Jul 31 13:13:09.159655 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.159575    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-fd6ea7\"/\"kube-root-ca.crt\""
Jul 31 13:13:09.159655 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.159603    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-fd6ea7\"/\"default-dockercfg-9v94g\""
Jul 31 13:13:09.159903 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.159720    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-fd6ea7\"/\"ipp-route-fd6ea7-kserve-self-signed-certs\""
Jul 31 13:13:09.166538 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.166512    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"]
Jul 31 13:13:09.223795 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.223757    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvlz\" (UniqueName: \"kubernetes.io/projected/9e73cdd1-6d18-4996-8f4d-815d8928926d-kube-api-access-5zvlz\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.223954 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.223842    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-home\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.223954 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.223871    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-dshm\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.224034 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.223957    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-model-cache\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.224034 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.223989    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e73cdd1-6d18-4996-8f4d-815d8928926d-tls-certs\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.224101 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.224051    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-tmp-dir\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.325248 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.325214    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvlz\" (UniqueName: \"kubernetes.io/projected/9e73cdd1-6d18-4996-8f4d-815d8928926d-kube-api-access-5zvlz\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.325420 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.325259    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-home\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.325420 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.325293    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-dshm\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.325420 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.325324    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-model-cache\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.325420 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.325340    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e73cdd1-6d18-4996-8f4d-815d8928926d-tls-certs\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.325605 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.325474    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-tmp-dir\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.325706 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.325689    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-home\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.325799 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.325779    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-model-cache\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.325889 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.325870    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-tmp-dir\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.327536 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.327514    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-dshm\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.327789 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.327772    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e73cdd1-6d18-4996-8f4d-815d8928926d-tls-certs\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.332646 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.332626    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvlz\" (UniqueName: \"kubernetes.io/projected/9e73cdd1-6d18-4996-8f4d-815d8928926d-kube-api-access-5zvlz\") pod \"ipp-route-fd6ea7-kserve-54b8bf6799-b4t65\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") " pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.467611 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.467518    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:09.606702 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.606680    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"]
Jul 31 13:13:09.608207 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:13:09.608181    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e73cdd1_6d18_4996_8f4d_815d8928926d.slice/crio-932bd9d4936551b52c964d2bfbe2fe2244c34a1edb97a2f3d1321ec71936b086 WatchSource:0}: Error finding container 932bd9d4936551b52c964d2bfbe2fe2244c34a1edb97a2f3d1321ec71936b086: Status 404 returned error can't find the container with id 932bd9d4936551b52c964d2bfbe2fe2244c34a1edb97a2f3d1321ec71936b086
Jul 31 13:13:09.642615 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:09.642585    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65" event={"ID":"9e73cdd1-6d18-4996-8f4d-815d8928926d","Type":"ContainerStarted","Data":"932bd9d4936551b52c964d2bfbe2fe2244c34a1edb97a2f3d1321ec71936b086"}
Jul 31 13:13:10.650194 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:10.650153    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65" event={"ID":"9e73cdd1-6d18-4996-8f4d-815d8928926d","Type":"ContainerStarted","Data":"56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9"}
Jul 31 13:13:10.667356 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:10.667306    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65" podStartSLOduration=1.66728888 podStartE2EDuration="1.66728888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:13:10.666384851 +0000 UTC m=+3478.653606392" watchObservedRunningTime="2026-07-31 13:13:10.66728888 +0000 UTC m=+3478.654510438"
Jul 31 13:13:19.468369 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:19.468332    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:19.468782 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:19.468394    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:19.481143 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:19.481118    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:19.710454 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:19.710424    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:13:47.690061 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:47.689978    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf_26497c12-1a75-42a9-8a48-c8fc230c25e4/payload-processing/0.log"
Jul 31 13:13:47.817879 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:47.817835    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pdrrk_d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2/payload-processing/0.log"
Jul 31 13:13:47.936165 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:47.936130    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5_b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2/payload-processing/0.log"
Jul 31 13:13:54.581207 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.581175    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"]
Jul 31 13:13:54.584278 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.584248    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:13:54.586723 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.586704    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2b382a-dockercfg-rzqzj\""
Jul 31 13:13:54.586848 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.586704    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2b382a\""
Jul 31 13:13:54.596967 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.596940    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"]
Jul 31 13:13:54.646259 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.646234    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r\" (UID: \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:13:54.646427 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.646339    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbmd\" (UniqueName: \"kubernetes.io/projected/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-kube-api-access-6vbmd\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r\" (UID: \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:13:54.747399 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.747358    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbmd\" (UniqueName: \"kubernetes.io/projected/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-kube-api-access-6vbmd\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r\" (UID: \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:13:54.747583 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.747472    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r\" (UID: \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:13:54.748122 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.748101    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r\" (UID: \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:13:54.755300 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.755259    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbmd\" (UniqueName: \"kubernetes.io/projected/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-kube-api-access-6vbmd\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r\" (UID: \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:13:54.895365 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:54.895280    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:13:55.025455 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:55.025430    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"]
Jul 31 13:13:55.893687 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:55.893647    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r" event={"ID":"4de3e33a-ac15-4db6-b35e-afc40cc8e6da","Type":"ContainerStarted","Data":"edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b"}
Jul 31 13:13:55.894071 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:55.893707    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r" event={"ID":"4de3e33a-ac15-4db6-b35e-afc40cc8e6da","Type":"ContainerStarted","Data":"19bd501adc57cd8ab9a9ea90c05de7a986669fa2c943f2f740c399fd837d067d"}
Jul 31 13:13:55.894071 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:55.893747    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:13:55.909596 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:13:55.909547    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r" podStartSLOduration=1.9095302159999998 podStartE2EDuration="1.909530216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:13:55.908378893 +0000 UTC m=+3523.895600436" watchObservedRunningTime="2026-07-31 13:13:55.909530216 +0000 UTC m=+3523.896751802"
Jul 31 13:14:06.901256 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:06.901227    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:14:11.920383 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:11.920345    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"]
Jul 31 13:14:11.923943 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:11.923926    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"
Jul 31 13:14:11.926565 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:11.926549    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lxtlm\""
Jul 31 13:14:11.932877 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:11.932855    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"]
Jul 31 13:14:12.024245 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:12.024213    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be10c300-0526-4ab1-8f5f-d72ab944e629-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs\" (UID: \"be10c300-0526-4ab1-8f5f-d72ab944e629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"
Jul 31 13:14:12.125826 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:12.125789    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be10c300-0526-4ab1-8f5f-d72ab944e629-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs\" (UID: \"be10c300-0526-4ab1-8f5f-d72ab944e629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"
Jul 31 13:14:12.126472 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:12.126454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be10c300-0526-4ab1-8f5f-d72ab944e629-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs\" (UID: \"be10c300-0526-4ab1-8f5f-d72ab944e629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"
Jul 31 13:14:12.236981 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:12.236895    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"
Jul 31 13:14:12.380501 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:12.380475    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"]
Jul 31 13:14:12.382243 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:14:12.382208    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe10c300_0526_4ab1_8f5f_d72ab944e629.slice/crio-7b9905612dc89e156494f313d619a5626a7964b5536c94a111d28ef3f4a9ac67 WatchSource:0}: Error finding container 7b9905612dc89e156494f313d619a5626a7964b5536c94a111d28ef3f4a9ac67: Status 404 returned error can't find the container with id 7b9905612dc89e156494f313d619a5626a7964b5536c94a111d28ef3f4a9ac67
Jul 31 13:14:12.993548 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:12.993512    2586 generic.go:356] "Generic (PLEG): container finished" podID="be10c300-0526-4ab1-8f5f-d72ab944e629" containerID="6d6c8e395f3664bd1ca964aff5e1bb99b8219ab17c30a52146f46d2f4cf7e6e8" exitCode=0
Jul 31 13:14:12.993912 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:12.993591    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs" event={"ID":"be10c300-0526-4ab1-8f5f-d72ab944e629","Type":"ContainerDied","Data":"6d6c8e395f3664bd1ca964aff5e1bb99b8219ab17c30a52146f46d2f4cf7e6e8"}
Jul 31 13:14:12.993912 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:12.993674    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs" event={"ID":"be10c300-0526-4ab1-8f5f-d72ab944e629","Type":"ContainerStarted","Data":"7b9905612dc89e156494f313d619a5626a7964b5536c94a111d28ef3f4a9ac67"}
Jul 31 13:14:14.142887 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:14.142865    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"
Jul 31 13:14:14.247276 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:14.247226    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be10c300-0526-4ab1-8f5f-d72ab944e629-maas-api-service-ca\") pod \"be10c300-0526-4ab1-8f5f-d72ab944e629\" (UID: \"be10c300-0526-4ab1-8f5f-d72ab944e629\") "
Jul 31 13:14:14.247576 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:14.247553    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be10c300-0526-4ab1-8f5f-d72ab944e629-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be10c300-0526-4ab1-8f5f-d72ab944e629" (UID: "be10c300-0526-4ab1-8f5f-d72ab944e629"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:14:14.348829 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:14.348762    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be10c300-0526-4ab1-8f5f-d72ab944e629-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:14:15.004743 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:15.004702    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs" event={"ID":"be10c300-0526-4ab1-8f5f-d72ab944e629","Type":"ContainerDied","Data":"7b9905612dc89e156494f313d619a5626a7964b5536c94a111d28ef3f4a9ac67"}
Jul 31 13:14:15.004743 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:15.004729    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"
Jul 31 13:14:15.004952 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:15.004763    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9905612dc89e156494f313d619a5626a7964b5536c94a111d28ef3f4a9ac67"
Jul 31 13:14:21.570116 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:21.570080    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"]
Jul 31 13:14:21.570739 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:21.570332    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r" podUID="4de3e33a-ac15-4db6-b35e-afc40cc8e6da" containerName="payload-pre-processing" containerID="cri-o://edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b" gracePeriod=30
Jul 31 13:14:21.702063 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:21.702033    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"]
Jul 31 13:14:21.710739 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:21.710712    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b382a-pf9bs"]
Jul 31 13:14:21.838615 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:21.838588    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:14:21.912192 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:21.912160    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-plugins-config-volume\") pod \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\" (UID: \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\") "
Jul 31 13:14:21.912192 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:21.912193    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vbmd\" (UniqueName: \"kubernetes.io/projected/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-kube-api-access-6vbmd\") pod \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\" (UID: \"4de3e33a-ac15-4db6-b35e-afc40cc8e6da\") "
Jul 31 13:14:21.912585 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:21.912557    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4de3e33a-ac15-4db6-b35e-afc40cc8e6da" (UID: "4de3e33a-ac15-4db6-b35e-afc40cc8e6da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:14:21.914242 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:21.914224    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-kube-api-access-6vbmd" (OuterVolumeSpecName: "kube-api-access-6vbmd") pod "4de3e33a-ac15-4db6-b35e-afc40cc8e6da" (UID: "4de3e33a-ac15-4db6-b35e-afc40cc8e6da"). InnerVolumeSpecName "kube-api-access-6vbmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:14:22.012977 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.012951    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:14:22.012977 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.012976    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vbmd\" (UniqueName: \"kubernetes.io/projected/4de3e33a-ac15-4db6-b35e-afc40cc8e6da-kube-api-access-6vbmd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:14:22.040401 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.040370    2586 generic.go:356] "Generic (PLEG): container finished" podID="4de3e33a-ac15-4db6-b35e-afc40cc8e6da" containerID="edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b" exitCode=0
Jul 31 13:14:22.040533 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.040423    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"
Jul 31 13:14:22.040533 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.040462    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r" event={"ID":"4de3e33a-ac15-4db6-b35e-afc40cc8e6da","Type":"ContainerDied","Data":"edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b"}
Jul 31 13:14:22.040533 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.040518    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r" event={"ID":"4de3e33a-ac15-4db6-b35e-afc40cc8e6da","Type":"ContainerDied","Data":"19bd501adc57cd8ab9a9ea90c05de7a986669fa2c943f2f740c399fd837d067d"}
Jul 31 13:14:22.040666 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.040540    2586 scope.go:117] "RemoveContainer" containerID="edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b"
Jul 31 13:14:22.050041 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.050026    2586 scope.go:117] "RemoveContainer" containerID="edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b"
Jul 31 13:14:22.050279 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:14:22.050246    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b\": container with ID starting with edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b not found: ID does not exist" containerID="edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b"
Jul 31 13:14:22.050332 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.050290    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b"} err="failed to get container status \"edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b\": rpc error: code = NotFound desc = could not find container \"edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b\": container with ID starting with edde77a46d0919dc4f3511e2f672094c1f4361ec97c29c03ec57e28bd1f7b52b not found: ID does not exist"
Jul 31 13:14:22.063242 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.063218    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"]
Jul 31 13:14:22.066748 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.066717    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b382a-74c479745b-nkr6r"]
Jul 31 13:14:22.485707 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.485673    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4de3e33a-ac15-4db6-b35e-afc40cc8e6da" path="/var/lib/kubelet/pods/4de3e33a-ac15-4db6-b35e-afc40cc8e6da/volumes"
Jul 31 13:14:22.486032 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:22.486019    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be10c300-0526-4ab1-8f5f-d72ab944e629" path="/var/lib/kubelet/pods/be10c300-0526-4ab1-8f5f-d72ab944e629/volumes"
Jul 31 13:14:49.312284 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.312235    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"]
Jul 31 13:14:49.312761 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.312510    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf" podUID="26497c12-1a75-42a9-8a48-c8fc230c25e4" containerName="payload-processing" containerID="cri-o://4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451" gracePeriod=30
Jul 31 13:14:49.319102 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.319075    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"]
Jul 31 13:14:49.319506 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.319458    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5" podUID="ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7" containerName="payload-pre-processing" containerID="cri-o://59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184" gracePeriod=30
Jul 31 13:14:49.665087 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.665060    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:14:49.667922 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.667900    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:14:49.782952 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.782920    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkd89\" (UniqueName: \"kubernetes.io/projected/26497c12-1a75-42a9-8a48-c8fc230c25e4-kube-api-access-qkd89\") pod \"26497c12-1a75-42a9-8a48-c8fc230c25e4\" (UID: \"26497c12-1a75-42a9-8a48-c8fc230c25e4\") "
Jul 31 13:14:49.783166 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.782995    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26497c12-1a75-42a9-8a48-c8fc230c25e4-plugins-config-volume\") pod \"26497c12-1a75-42a9-8a48-c8fc230c25e4\" (UID: \"26497c12-1a75-42a9-8a48-c8fc230c25e4\") "
Jul 31 13:14:49.783166 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.783095    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-plugins-config-volume\") pod \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\" (UID: \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\") "
Jul 31 13:14:49.783166 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.783135    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwr9z\" (UniqueName: \"kubernetes.io/projected/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-kube-api-access-hwr9z\") pod \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\" (UID: \"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7\") "
Jul 31 13:14:49.783483 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.783442    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26497c12-1a75-42a9-8a48-c8fc230c25e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26497c12-1a75-42a9-8a48-c8fc230c25e4" (UID: "26497c12-1a75-42a9-8a48-c8fc230c25e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:14:49.783538 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.783487    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7" (UID: "ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:14:49.785164 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.785129    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26497c12-1a75-42a9-8a48-c8fc230c25e4-kube-api-access-qkd89" (OuterVolumeSpecName: "kube-api-access-qkd89") pod "26497c12-1a75-42a9-8a48-c8fc230c25e4" (UID: "26497c12-1a75-42a9-8a48-c8fc230c25e4"). InnerVolumeSpecName "kube-api-access-qkd89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:14:49.785397 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.785169    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-kube-api-access-hwr9z" (OuterVolumeSpecName: "kube-api-access-hwr9z") pod "ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7" (UID: "ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7"). InnerVolumeSpecName "kube-api-access-hwr9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:14:49.884258 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.884222    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26497c12-1a75-42a9-8a48-c8fc230c25e4-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:14:49.884258 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.884248    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:14:49.884258 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.884258    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwr9z\" (UniqueName: \"kubernetes.io/projected/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7-kube-api-access-hwr9z\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:14:49.884258 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:49.884296    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkd89\" (UniqueName: \"kubernetes.io/projected/26497c12-1a75-42a9-8a48-c8fc230c25e4-kube-api-access-qkd89\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:14:50.198258 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.198226    2586 generic.go:356] "Generic (PLEG): container finished" podID="26497c12-1a75-42a9-8a48-c8fc230c25e4" containerID="4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451" exitCode=0
Jul 31 13:14:50.198448 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.198283    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"
Jul 31 13:14:50.198448 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.198320    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf" event={"ID":"26497c12-1a75-42a9-8a48-c8fc230c25e4","Type":"ContainerDied","Data":"4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451"}
Jul 31 13:14:50.198448 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.198413    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf" event={"ID":"26497c12-1a75-42a9-8a48-c8fc230c25e4","Type":"ContainerDied","Data":"22c27bfec2c8fe946a7cf19d3bec4e3e5f0368c4f8ed5afd6034526d9e1c34a6"}
Jul 31 13:14:50.198448 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.198444    2586 scope.go:117] "RemoveContainer" containerID="4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451"
Jul 31 13:14:50.199936 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.199913    2586 generic.go:356] "Generic (PLEG): container finished" podID="ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7" containerID="59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184" exitCode=0
Jul 31 13:14:50.200049 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.199955    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"
Jul 31 13:14:50.200049 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.200003    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5" event={"ID":"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7","Type":"ContainerDied","Data":"59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184"}
Jul 31 13:14:50.200167 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.200056    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5" event={"ID":"ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7","Type":"ContainerDied","Data":"bd9fe262fd702346c7795566152fb8d7e204b9d5d613652d5e497d6d08b01d99"}
Jul 31 13:14:50.208832 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.208809    2586 scope.go:117] "RemoveContainer" containerID="4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451"
Jul 31 13:14:50.209099 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:14:50.209081    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451\": container with ID starting with 4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451 not found: ID does not exist" containerID="4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451"
Jul 31 13:14:50.209151 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.209107    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451"} err="failed to get container status \"4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451\": rpc error: code = NotFound desc = could not find container \"4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451\": container with ID starting with 4f509961b357eacac85b42410f27814f93b86a324744591d6d9624fab4199451 not found: ID does not exist"
Jul 31 13:14:50.209151 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.209125    2586 scope.go:117] "RemoveContainer" containerID="59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184"
Jul 31 13:14:50.217189 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.217173    2586 scope.go:117] "RemoveContainer" containerID="59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184"
Jul 31 13:14:50.218160 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:14:50.218142    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184\": container with ID starting with 59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184 not found: ID does not exist" containerID="59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184"
Jul 31 13:14:50.218256 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.218165    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184"} err="failed to get container status \"59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184\": rpc error: code = NotFound desc = could not find container \"59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184\": container with ID starting with 59f81662e1b6134a806f83c5d2fcb93322dca0fae6d0da333d660fcfbcf42184 not found: ID does not exist"
Jul 31 13:14:50.223064 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.223043    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"]
Jul 31 13:14:50.226771 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.226752    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd6ea7-bf5cd6644-r5jt5"]
Jul 31 13:14:50.236981 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.236961    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"]
Jul 31 13:14:50.238883 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.238866    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd6ea7-7b7fdc8cb8-sjxkf"]
Jul 31 13:14:50.485699 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.485606    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26497c12-1a75-42a9-8a48-c8fc230c25e4" path="/var/lib/kubelet/pods/26497c12-1a75-42a9-8a48-c8fc230c25e4/volumes"
Jul 31 13:14:50.486206 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:50.486157    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7" path="/var/lib/kubelet/pods/ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7/volumes"
Jul 31 13:14:54.811755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:14:54.811722    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"]
Jul 31 13:15:03.993784 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:03.993746    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql"]
Jul 31 13:15:03.995918 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:03.995895    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758350-x4cql"]
Jul 31 13:15:04.484819 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:04.484784    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a4c00e-0358-4a0a-92ed-847414676cbb" path="/var/lib/kubelet/pods/85a4c00e-0358-4a0a-92ed-847414676cbb/volumes"
Jul 31 13:15:09.831175 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:09.831134    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65" podUID="9e73cdd1-6d18-4996-8f4d-815d8928926d" containerName="main" containerID="cri-o://56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9" gracePeriod=45
Jul 31 13:15:10.097720 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.097655    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:15:10.163494 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.163463    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-home\") pod \"9e73cdd1-6d18-4996-8f4d-815d8928926d\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") "
Jul 31 13:15:10.163683 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.163568    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zvlz\" (UniqueName: \"kubernetes.io/projected/9e73cdd1-6d18-4996-8f4d-815d8928926d-kube-api-access-5zvlz\") pod \"9e73cdd1-6d18-4996-8f4d-815d8928926d\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") "
Jul 31 13:15:10.163683 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.163604    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-model-cache\") pod \"9e73cdd1-6d18-4996-8f4d-815d8928926d\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") "
Jul 31 13:15:10.163683 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.163629    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-tmp-dir\") pod \"9e73cdd1-6d18-4996-8f4d-815d8928926d\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") "
Jul 31 13:15:10.163683 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.163653    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-dshm\") pod \"9e73cdd1-6d18-4996-8f4d-815d8928926d\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") "
Jul 31 13:15:10.163683 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.163678    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e73cdd1-6d18-4996-8f4d-815d8928926d-tls-certs\") pod \"9e73cdd1-6d18-4996-8f4d-815d8928926d\" (UID: \"9e73cdd1-6d18-4996-8f4d-815d8928926d\") "
Jul 31 13:15:10.163951 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.163773    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-home" (OuterVolumeSpecName: "home") pod "9e73cdd1-6d18-4996-8f4d-815d8928926d" (UID: "9e73cdd1-6d18-4996-8f4d-815d8928926d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:15:10.163951 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.163885    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e73cdd1-6d18-4996-8f4d-815d8928926d" (UID: "9e73cdd1-6d18-4996-8f4d-815d8928926d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:15:10.163951 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.163912    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e73cdd1-6d18-4996-8f4d-815d8928926d" (UID: "9e73cdd1-6d18-4996-8f4d-815d8928926d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:15:10.164115 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.164039    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:10.164115 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.164057    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:10.164115 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.164073    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:10.165950 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.165920    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e73cdd1-6d18-4996-8f4d-815d8928926d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e73cdd1-6d18-4996-8f4d-815d8928926d" (UID: "9e73cdd1-6d18-4996-8f4d-815d8928926d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 13:15:10.166216 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.166194    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e73cdd1-6d18-4996-8f4d-815d8928926d-kube-api-access-5zvlz" (OuterVolumeSpecName: "kube-api-access-5zvlz") pod "9e73cdd1-6d18-4996-8f4d-815d8928926d" (UID: "9e73cdd1-6d18-4996-8f4d-815d8928926d"). InnerVolumeSpecName "kube-api-access-5zvlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:15:10.166349 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.166236    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-dshm" (OuterVolumeSpecName: "dshm") pod "9e73cdd1-6d18-4996-8f4d-815d8928926d" (UID: "9e73cdd1-6d18-4996-8f4d-815d8928926d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:15:10.264737 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.264709    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e73cdd1-6d18-4996-8f4d-815d8928926d-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:10.264737 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.264735    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zvlz\" (UniqueName: \"kubernetes.io/projected/9e73cdd1-6d18-4996-8f4d-815d8928926d-kube-api-access-5zvlz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:10.264737 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.264744    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e73cdd1-6d18-4996-8f4d-815d8928926d-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:10.308088 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.308050    2586 generic.go:356] "Generic (PLEG): container finished" podID="9e73cdd1-6d18-4996-8f4d-815d8928926d" containerID="56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9" exitCode=0
Jul 31 13:15:10.308088 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.308092    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"
Jul 31 13:15:10.308324 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.308137    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65" event={"ID":"9e73cdd1-6d18-4996-8f4d-815d8928926d","Type":"ContainerDied","Data":"56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9"}
Jul 31 13:15:10.308324 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.308185    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65" event={"ID":"9e73cdd1-6d18-4996-8f4d-815d8928926d","Type":"ContainerDied","Data":"932bd9d4936551b52c964d2bfbe2fe2244c34a1edb97a2f3d1321ec71936b086"}
Jul 31 13:15:10.308324 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.308206    2586 scope.go:117] "RemoveContainer" containerID="56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9"
Jul 31 13:15:10.317765 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.317744    2586 scope.go:117] "RemoveContainer" containerID="56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9"
Jul 31 13:15:10.318069 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:15:10.318045    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9\": container with ID starting with 56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9 not found: ID does not exist" containerID="56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9"
Jul 31 13:15:10.318165 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.318075    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9"} err="failed to get container status \"56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9\": rpc error: code = NotFound desc = could not find container \"56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9\": container with ID starting with 56f619b9f1a43c247b8b6ceacfa12ab01af4bb3e21b49f2eda67bf34040658d9 not found: ID does not exist"
Jul 31 13:15:10.328968 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.328944    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"]
Jul 31 13:15:10.330988 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.330966    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-fd6ea7/ipp-route-fd6ea7-kserve-54b8bf6799-b4t65"]
Jul 31 13:15:10.485340 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:10.485306    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e73cdd1-6d18-4996-8f4d-815d8928926d" path="/var/lib/kubelet/pods/9e73cdd1-6d18-4996-8f4d-815d8928926d/volumes"
Jul 31 13:15:12.954137 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:12.954105    2586 scope.go:117] "RemoveContainer" containerID="aa342784df5370f574d1f903539bfc0ace5158f603289ea4e6b393353cfdbee7"
Jul 31 13:15:38.958165 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958130    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"]
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958546    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de3e33a-ac15-4db6-b35e-afc40cc8e6da" containerName="payload-pre-processing"
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958557    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de3e33a-ac15-4db6-b35e-afc40cc8e6da" containerName="payload-pre-processing"
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958570    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7" containerName="payload-pre-processing"
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958578    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7" containerName="payload-pre-processing"
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958590    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26497c12-1a75-42a9-8a48-c8fc230c25e4" containerName="payload-processing"
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958596    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="26497c12-1a75-42a9-8a48-c8fc230c25e4" containerName="payload-processing"
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958605    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be10c300-0526-4ab1-8f5f-d72ab944e629" containerName="revoke-keys"
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958610    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="be10c300-0526-4ab1-8f5f-d72ab944e629" containerName="revoke-keys"
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958620    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e73cdd1-6d18-4996-8f4d-815d8928926d" containerName="main"
Jul 31 13:15:38.958650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958625    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e73cdd1-6d18-4996-8f4d-815d8928926d" containerName="main"
Jul 31 13:15:38.958959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958678    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead09e73-2bf2-4bc7-93c9-d28eae8ca5e7" containerName="payload-pre-processing"
Jul 31 13:15:38.958959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958686    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="26497c12-1a75-42a9-8a48-c8fc230c25e4" containerName="payload-processing"
Jul 31 13:15:38.958959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958693    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e73cdd1-6d18-4996-8f4d-815d8928926d" containerName="main"
Jul 31 13:15:38.958959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958699    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de3e33a-ac15-4db6-b35e-afc40cc8e6da" containerName="payload-pre-processing"
Jul 31 13:15:38.958959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.958705    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="be10c300-0526-4ab1-8f5f-d72ab944e629" containerName="revoke-keys"
Jul 31 13:15:38.961621 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.961605    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"
Jul 31 13:15:38.964927 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.964903    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lxtlm\""
Jul 31 13:15:38.971568 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:38.971534    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"]
Jul 31 13:15:39.139396 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:39.139362    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bea8e2-0df3-4146-9ba0-89bc83b16f82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk\" (UID: \"c6bea8e2-0df3-4146-9ba0-89bc83b16f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"
Jul 31 13:15:39.240225 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:39.240124    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bea8e2-0df3-4146-9ba0-89bc83b16f82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk\" (UID: \"c6bea8e2-0df3-4146-9ba0-89bc83b16f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"
Jul 31 13:15:39.240755 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:39.240734    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bea8e2-0df3-4146-9ba0-89bc83b16f82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk\" (UID: \"c6bea8e2-0df3-4146-9ba0-89bc83b16f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"
Jul 31 13:15:39.273130 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:39.273101    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"
Jul 31 13:15:39.411654 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:39.411628    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"]
Jul 31 13:15:39.413992 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:15:39.413964    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bea8e2_0df3_4146_9ba0_89bc83b16f82.slice/crio-ba25d06b01b00461b92f3ba76702b8191d876490767cb07fb1e8f82309ce612d WatchSource:0}: Error finding container ba25d06b01b00461b92f3ba76702b8191d876490767cb07fb1e8f82309ce612d: Status 404 returned error can't find the container with id ba25d06b01b00461b92f3ba76702b8191d876490767cb07fb1e8f82309ce612d
Jul 31 13:15:39.454313 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:39.454286    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk" event={"ID":"c6bea8e2-0df3-4146-9ba0-89bc83b16f82","Type":"ContainerStarted","Data":"ba25d06b01b00461b92f3ba76702b8191d876490767cb07fb1e8f82309ce612d"}
Jul 31 13:15:40.460941 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:40.460907    2586 generic.go:356] "Generic (PLEG): container finished" podID="c6bea8e2-0df3-4146-9ba0-89bc83b16f82" containerID="217d99c7ce561d337297bf19c1b23164efc033fcebabef86bfa88e27e9832a9b" exitCode=0
Jul 31 13:15:40.461404 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:40.460997    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk" event={"ID":"c6bea8e2-0df3-4146-9ba0-89bc83b16f82","Type":"ContainerDied","Data":"217d99c7ce561d337297bf19c1b23164efc033fcebabef86bfa88e27e9832a9b"}
Jul 31 13:15:41.601631 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:41.601609    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"
Jul 31 13:15:41.763744 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:41.763650    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bea8e2-0df3-4146-9ba0-89bc83b16f82-maas-api-service-ca\") pod \"c6bea8e2-0df3-4146-9ba0-89bc83b16f82\" (UID: \"c6bea8e2-0df3-4146-9ba0-89bc83b16f82\") "
Jul 31 13:15:41.763982 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:41.763960    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6bea8e2-0df3-4146-9ba0-89bc83b16f82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6bea8e2-0df3-4146-9ba0-89bc83b16f82" (UID: "c6bea8e2-0df3-4146-9ba0-89bc83b16f82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:15:41.864702 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:41.864668    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bea8e2-0df3-4146-9ba0-89bc83b16f82-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:42.471709 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:42.471679    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"
Jul 31 13:15:42.471893 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:42.471725    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk" event={"ID":"c6bea8e2-0df3-4146-9ba0-89bc83b16f82","Type":"ContainerDied","Data":"ba25d06b01b00461b92f3ba76702b8191d876490767cb07fb1e8f82309ce612d"}
Jul 31 13:15:42.471893 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:42.471814    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba25d06b01b00461b92f3ba76702b8191d876490767cb07fb1e8f82309ce612d"
Jul 31 13:15:46.264909 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.264876    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"]
Jul 31 13:15:46.265500 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.265076    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5" podUID="b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2" containerName="payload-processing" containerID="cri-o://92e775be20a94096bfa59e91af8018efd6c1d95aa82634863f165c6ad74b0e69" gracePeriod=30
Jul 31 13:15:46.492485 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.492454    2586 generic.go:356] "Generic (PLEG): container finished" podID="b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2" containerID="92e775be20a94096bfa59e91af8018efd6c1d95aa82634863f165c6ad74b0e69" exitCode=0
Jul 31 13:15:46.492650 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.492528    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5" event={"ID":"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2","Type":"ContainerDied","Data":"92e775be20a94096bfa59e91af8018efd6c1d95aa82634863f165c6ad74b0e69"}
Jul 31 13:15:46.519219 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.519165    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:15:46.560928 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.560898    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"]
Jul 31 13:15:46.569190 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.569041    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f59489-mqbzk"]
Jul 31 13:15:46.610684 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.610655    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-plugins-config-volume\") pod \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\" (UID: \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\") "
Jul 31 13:15:46.610856 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.610757    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzhm9\" (UniqueName: \"kubernetes.io/projected/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-kube-api-access-mzhm9\") pod \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\" (UID: \"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2\") "
Jul 31 13:15:46.611081 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.611057    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2" (UID: "b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:15:46.612766 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.612741    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-kube-api-access-mzhm9" (OuterVolumeSpecName: "kube-api-access-mzhm9") pod "b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2" (UID: "b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2"). InnerVolumeSpecName "kube-api-access-mzhm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:15:46.712132 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.712101    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzhm9\" (UniqueName: \"kubernetes.io/projected/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-kube-api-access-mzhm9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:46.712132 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:46.712127    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:47.498988 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:47.498957    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"
Jul 31 13:15:47.499425 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:47.498984    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5" event={"ID":"b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2","Type":"ContainerDied","Data":"e8f98218d4714b69906482bab373af3fb34e5d0cfa22d35a6cb7e6c18ce1e41c"}
Jul 31 13:15:47.499425 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:47.499044    2586 scope.go:117] "RemoveContainer" containerID="92e775be20a94096bfa59e91af8018efd6c1d95aa82634863f165c6ad74b0e69"
Jul 31 13:15:47.521451 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:47.521427    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"]
Jul 31 13:15:47.526906 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:47.526885    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f59489-5497f4c946-mf7n5"]
Jul 31 13:15:48.485220 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:48.485187    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2" path="/var/lib/kubelet/pods/b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2/volumes"
Jul 31 13:15:48.485549 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:48.485536    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6bea8e2-0df3-4146-9ba0-89bc83b16f82" path="/var/lib/kubelet/pods/c6bea8e2-0df3-4146-9ba0-89bc83b16f82/volumes"
Jul 31 13:15:53.550952 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.550921    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"]
Jul 31 13:15:53.551351 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.551337    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6bea8e2-0df3-4146-9ba0-89bc83b16f82" containerName="revoke-keys"
Jul 31 13:15:53.551351 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.551352    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6bea8e2-0df3-4146-9ba0-89bc83b16f82" containerName="revoke-keys"
Jul 31 13:15:53.551473 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.551369    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2" containerName="payload-processing"
Jul 31 13:15:53.551473 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.551376    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2" containerName="payload-processing"
Jul 31 13:15:53.551473 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.551464    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6bea8e2-0df3-4146-9ba0-89bc83b16f82" containerName="revoke-keys"
Jul 31 13:15:53.551591 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.551478    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b92e1cf0-af4e-4d22-bfb8-57b2c01a51f2" containerName="payload-processing"
Jul 31 13:15:53.555858 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.555841    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"
Jul 31 13:15:53.558339 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.558313    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lxtlm\""
Jul 31 13:15:53.561167 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.561137    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"]
Jul 31 13:15:53.574007 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.573984    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02cd63-0cc4-40b3-8f5f-6e743988f358-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw\" (UID: \"be02cd63-0cc4-40b3-8f5f-6e743988f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"
Jul 31 13:15:53.675195 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.675165    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02cd63-0cc4-40b3-8f5f-6e743988f358-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw\" (UID: \"be02cd63-0cc4-40b3-8f5f-6e743988f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"
Jul 31 13:15:53.675786 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.675766    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02cd63-0cc4-40b3-8f5f-6e743988f358-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw\" (UID: \"be02cd63-0cc4-40b3-8f5f-6e743988f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"
Jul 31 13:15:53.868483 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:53.868403    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"
Jul 31 13:15:54.004206 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:54.004176    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"]
Jul 31 13:15:54.005063 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:15:54.005023    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe02cd63_0cc4_40b3_8f5f_6e743988f358.slice/crio-b34723334a662d2bbe11a63391148aac7afa3728fd942169a122c4c69e08cbd2 WatchSource:0}: Error finding container b34723334a662d2bbe11a63391148aac7afa3728fd942169a122c4c69e08cbd2: Status 404 returned error can't find the container with id b34723334a662d2bbe11a63391148aac7afa3728fd942169a122c4c69e08cbd2
Jul 31 13:15:54.536687 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:54.536609    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw" event={"ID":"be02cd63-0cc4-40b3-8f5f-6e743988f358","Type":"ContainerStarted","Data":"80c6de88da984185161053946100c81e52b797543101adfac79f063313f97ca9"}
Jul 31 13:15:54.536687 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:54.536664    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw" event={"ID":"be02cd63-0cc4-40b3-8f5f-6e743988f358","Type":"ContainerStarted","Data":"b34723334a662d2bbe11a63391148aac7afa3728fd942169a122c4c69e08cbd2"}
Jul 31 13:15:55.543228 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:55.543196    2586 generic.go:356] "Generic (PLEG): container finished" podID="be02cd63-0cc4-40b3-8f5f-6e743988f358" containerID="80c6de88da984185161053946100c81e52b797543101adfac79f063313f97ca9" exitCode=0
Jul 31 13:15:55.543669 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:55.543295    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw" event={"ID":"be02cd63-0cc4-40b3-8f5f-6e743988f358","Type":"ContainerDied","Data":"80c6de88da984185161053946100c81e52b797543101adfac79f063313f97ca9"}
Jul 31 13:15:56.684906 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:56.684880    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"
Jul 31 13:15:56.806893 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:56.806806    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02cd63-0cc4-40b3-8f5f-6e743988f358-maas-api-service-ca\") pod \"be02cd63-0cc4-40b3-8f5f-6e743988f358\" (UID: \"be02cd63-0cc4-40b3-8f5f-6e743988f358\") "
Jul 31 13:15:56.807173 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:56.807150    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be02cd63-0cc4-40b3-8f5f-6e743988f358-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be02cd63-0cc4-40b3-8f5f-6e743988f358" (UID: "be02cd63-0cc4-40b3-8f5f-6e743988f358"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:15:56.907990 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:56.907960    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02cd63-0cc4-40b3-8f5f-6e743988f358-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:15:57.555122 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:57.555094    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"
Jul 31 13:15:57.555122 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:57.555107    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw" event={"ID":"be02cd63-0cc4-40b3-8f5f-6e743988f358","Type":"ContainerDied","Data":"b34723334a662d2bbe11a63391148aac7afa3728fd942169a122c4c69e08cbd2"}
Jul 31 13:15:57.555355 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:15:57.555151    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b34723334a662d2bbe11a63391148aac7afa3728fd942169a122c4c69e08cbd2"
Jul 31 13:16:00.287478 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.287441    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"]
Jul 31 13:16:00.287935 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.287718    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2" podUID="01fd4400-bb38-4f3a-9eb7-4119889c64fa" containerName="maas-api" containerID="cri-o://a7a5ee46df57d4846566ec2d35f0188cc3a60d593bc9170edd131f8b242037bc" gracePeriod=30
Jul 31 13:16:00.420972 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.420931    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2" podUID="01fd4400-bb38-4f3a-9eb7-4119889c64fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.83:8443/health\": dial tcp 10.132.0.83:8443: connect: connection refused"
Jul 31 13:16:00.489112 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.489083    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"]
Jul 31 13:16:00.489363 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.489136    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"]
Jul 31 13:16:00.489623 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.489561    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294" podUID="558ec461-aa59-4217-a669-309c134b7664" containerName="payload-processing" containerID="cri-o://3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6" gracePeriod=30
Jul 31 13:16:00.489900 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.489878    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q" podUID="931d1041-1a96-407f-a1ff-dac0e9a294bf" containerName="payload-pre-processing" containerID="cri-o://5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f" gracePeriod=30
Jul 31 13:16:00.571589 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.571556    2586 generic.go:356] "Generic (PLEG): container finished" podID="01fd4400-bb38-4f3a-9eb7-4119889c64fa" containerID="a7a5ee46df57d4846566ec2d35f0188cc3a60d593bc9170edd131f8b242037bc" exitCode=0
Jul 31 13:16:00.571705 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.571625    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2" event={"ID":"01fd4400-bb38-4f3a-9eb7-4119889c64fa","Type":"ContainerDied","Data":"a7a5ee46df57d4846566ec2d35f0188cc3a60d593bc9170edd131f8b242037bc"}
Jul 31 13:16:00.627389 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.627198    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:16:00.664388 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.663621    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"]
Jul 31 13:16:00.667022 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.666937    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-96b3e0-hp4jw"]
Jul 31 13:16:00.748849 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.748823    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjkgt\" (UniqueName: \"kubernetes.io/projected/01fd4400-bb38-4f3a-9eb7-4119889c64fa-kube-api-access-bjkgt\") pod \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") "
Jul 31 13:16:00.748995 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.748947    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fd4400-bb38-4f3a-9eb7-4119889c64fa-openshift-service-ca\") pod \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") "
Jul 31 13:16:00.748995 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.748969    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls\") pod \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\" (UID: \"01fd4400-bb38-4f3a-9eb7-4119889c64fa\") "
Jul 31 13:16:00.749365 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.749338    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fd4400-bb38-4f3a-9eb7-4119889c64fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01fd4400-bb38-4f3a-9eb7-4119889c64fa" (UID: "01fd4400-bb38-4f3a-9eb7-4119889c64fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:16:00.751961 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.751937    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01fd4400-bb38-4f3a-9eb7-4119889c64fa" (UID: "01fd4400-bb38-4f3a-9eb7-4119889c64fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 13:16:00.753506 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.753481    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fd4400-bb38-4f3a-9eb7-4119889c64fa-kube-api-access-bjkgt" (OuterVolumeSpecName: "kube-api-access-bjkgt") pod "01fd4400-bb38-4f3a-9eb7-4119889c64fa" (UID: "01fd4400-bb38-4f3a-9eb7-4119889c64fa"). InnerVolumeSpecName "kube-api-access-bjkgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:16:00.783575 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.783557    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:16:00.786580 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.786562    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:16:00.849744 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.849688    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558ec461-aa59-4217-a669-309c134b7664-plugins-config-volume\") pod \"558ec461-aa59-4217-a669-309c134b7664\" (UID: \"558ec461-aa59-4217-a669-309c134b7664\") "
Jul 31 13:16:00.849744 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.849740    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gdgg\" (UniqueName: \"kubernetes.io/projected/931d1041-1a96-407f-a1ff-dac0e9a294bf-kube-api-access-9gdgg\") pod \"931d1041-1a96-407f-a1ff-dac0e9a294bf\" (UID: \"931d1041-1a96-407f-a1ff-dac0e9a294bf\") "
Jul 31 13:16:00.849934 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.849768    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d1041-1a96-407f-a1ff-dac0e9a294bf-plugins-config-volume\") pod \"931d1041-1a96-407f-a1ff-dac0e9a294bf\" (UID: \"931d1041-1a96-407f-a1ff-dac0e9a294bf\") "
Jul 31 13:16:00.849934 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.849834    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z5r4\" (UniqueName: \"kubernetes.io/projected/558ec461-aa59-4217-a669-309c134b7664-kube-api-access-5z5r4\") pod \"558ec461-aa59-4217-a669-309c134b7664\" (UID: \"558ec461-aa59-4217-a669-309c134b7664\") "
Jul 31 13:16:00.850070 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.850048    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558ec461-aa59-4217-a669-309c134b7664-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "558ec461-aa59-4217-a669-309c134b7664" (UID: "558ec461-aa59-4217-a669-309c134b7664"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:16:00.850136 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.850080    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fd4400-bb38-4f3a-9eb7-4119889c64fa-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:00.850136 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.850100    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fd4400-bb38-4f3a-9eb7-4119889c64fa-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:00.850136 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.850114    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjkgt\" (UniqueName: \"kubernetes.io/projected/01fd4400-bb38-4f3a-9eb7-4119889c64fa-kube-api-access-bjkgt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:00.850259 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.850157    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931d1041-1a96-407f-a1ff-dac0e9a294bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "931d1041-1a96-407f-a1ff-dac0e9a294bf" (UID: "931d1041-1a96-407f-a1ff-dac0e9a294bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:16:00.851685 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.851667    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931d1041-1a96-407f-a1ff-dac0e9a294bf-kube-api-access-9gdgg" (OuterVolumeSpecName: "kube-api-access-9gdgg") pod "931d1041-1a96-407f-a1ff-dac0e9a294bf" (UID: "931d1041-1a96-407f-a1ff-dac0e9a294bf"). InnerVolumeSpecName "kube-api-access-9gdgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:16:00.851808 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.851791    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558ec461-aa59-4217-a669-309c134b7664-kube-api-access-5z5r4" (OuterVolumeSpecName: "kube-api-access-5z5r4") pod "558ec461-aa59-4217-a669-309c134b7664" (UID: "558ec461-aa59-4217-a669-309c134b7664"). InnerVolumeSpecName "kube-api-access-5z5r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:16:00.950613 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.950583    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z5r4\" (UniqueName: \"kubernetes.io/projected/558ec461-aa59-4217-a669-309c134b7664-kube-api-access-5z5r4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:00.950613 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.950607    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558ec461-aa59-4217-a669-309c134b7664-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:00.950613 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.950617    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gdgg\" (UniqueName: \"kubernetes.io/projected/931d1041-1a96-407f-a1ff-dac0e9a294bf-kube-api-access-9gdgg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:00.950820 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:00.950625    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d1041-1a96-407f-a1ff-dac0e9a294bf-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:01.577981 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.577890    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"
Jul 31 13:16:01.577981 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.577890    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2" event={"ID":"01fd4400-bb38-4f3a-9eb7-4119889c64fa","Type":"ContainerDied","Data":"e084afdfefa965bf996642f230d42ef40fe792747079239d4d3a6c7dea4b1bb5"}
Jul 31 13:16:01.578524 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.578021    2586 scope.go:117] "RemoveContainer" containerID="a7a5ee46df57d4846566ec2d35f0188cc3a60d593bc9170edd131f8b242037bc"
Jul 31 13:16:01.579477 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.579235    2586 generic.go:356] "Generic (PLEG): container finished" podID="558ec461-aa59-4217-a669-309c134b7664" containerID="3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6" exitCode=0
Jul 31 13:16:01.579477 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.579295    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"
Jul 31 13:16:01.579477 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.579300    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294" event={"ID":"558ec461-aa59-4217-a669-309c134b7664","Type":"ContainerDied","Data":"3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6"}
Jul 31 13:16:01.579477 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.579367    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294" event={"ID":"558ec461-aa59-4217-a669-309c134b7664","Type":"ContainerDied","Data":"e4b868f062ea7d697f761f6299165e70d620d16c8ce1a40193e0cc97aa866a32"}
Jul 31 13:16:01.580842 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.580818    2586 generic.go:356] "Generic (PLEG): container finished" podID="931d1041-1a96-407f-a1ff-dac0e9a294bf" containerID="5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f" exitCode=0
Jul 31 13:16:01.580957 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.580852    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"
Jul 31 13:16:01.580957 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.580858    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q" event={"ID":"931d1041-1a96-407f-a1ff-dac0e9a294bf","Type":"ContainerDied","Data":"5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f"}
Jul 31 13:16:01.580957 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.580906    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q" event={"ID":"931d1041-1a96-407f-a1ff-dac0e9a294bf","Type":"ContainerDied","Data":"a7d1d703c03540891ccc48c538caffcc07e0676a1de3d2ac3a57fe3d4f532204"}
Jul 31 13:16:01.588352 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.588219    2586 scope.go:117] "RemoveContainer" containerID="3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6"
Jul 31 13:16:01.597568 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.597554    2586 scope.go:117] "RemoveContainer" containerID="3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6"
Jul 31 13:16:01.597830 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:16:01.597810    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6\": container with ID starting with 3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6 not found: ID does not exist" containerID="3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6"
Jul 31 13:16:01.597915 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.597841    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6"} err="failed to get container status \"3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6\": rpc error: code = NotFound desc = could not find container \"3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6\": container with ID starting with 3df5e4c70ef490de377091cdd7c0e7bee0c765db4ed0179a1f10fabdf47689e6 not found: ID does not exist"
Jul 31 13:16:01.597915 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.597865    2586 scope.go:117] "RemoveContainer" containerID="5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f"
Jul 31 13:16:01.605228 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.605210    2586 scope.go:117] "RemoveContainer" containerID="5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f"
Jul 31 13:16:01.605498 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:16:01.605476    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f\": container with ID starting with 5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f not found: ID does not exist" containerID="5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f"
Jul 31 13:16:01.605607 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.605502    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f"} err="failed to get container status \"5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f\": rpc error: code = NotFound desc = could not find container \"5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f\": container with ID starting with 5bb3a79e466f315bd6723231c23f714a5b63586633758bde4e31741f6ba5a56f not found: ID does not exist"
Jul 31 13:16:01.607524 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.607507    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"]
Jul 31 13:16:01.611526 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.611505    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-96b3e0-85cfcd4b65-zs9n2"]
Jul 31 13:16:01.623455 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.623433    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"]
Jul 31 13:16:01.625124 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.625106    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-96b3e0-5b8dbcf6f6-xls6q"]
Jul 31 13:16:01.636248 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.636229    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"]
Jul 31 13:16:01.639943 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:01.639925    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-96b3e0-ccc654df8-tf294"]
Jul 31 13:16:02.485807 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:02.485777    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01fd4400-bb38-4f3a-9eb7-4119889c64fa" path="/var/lib/kubelet/pods/01fd4400-bb38-4f3a-9eb7-4119889c64fa/volumes"
Jul 31 13:16:02.486231 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:02.486214    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558ec461-aa59-4217-a669-309c134b7664" path="/var/lib/kubelet/pods/558ec461-aa59-4217-a669-309c134b7664/volumes"
Jul 31 13:16:02.486623 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:02.486607    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="931d1041-1a96-407f-a1ff-dac0e9a294bf" path="/var/lib/kubelet/pods/931d1041-1a96-407f-a1ff-dac0e9a294bf/volumes"
Jul 31 13:16:02.486922 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:02.486910    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be02cd63-0cc4-40b3-8f5f-6e743988f358" path="/var/lib/kubelet/pods/be02cd63-0cc4-40b3-8f5f-6e743988f358/volumes"
Jul 31 13:16:07.364422 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:07.364384    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"]
Jul 31 13:16:07.365190 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:07.365155    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy" containerID="cri-o://b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38" gracePeriod=30
Jul 31 13:16:10.789168 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:10.789128    2586 patch_prober.go:28] interesting pod/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 13:16:10.789569 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:10.789197    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 13:16:12.627918 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.627895    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:16:12.640933 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.640871    2586 generic.go:356] "Generic (PLEG): container finished" podID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerID="b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38" exitCode=0
Jul 31 13:16:12.640933 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.640926    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"
Jul 31 13:16:12.641118 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.640951    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" event={"ID":"ba4af621-bf34-49a3-8870-ecbd873d50b2","Type":"ContainerDied","Data":"b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38"}
Jul 31 13:16:12.641118 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.640998    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k" event={"ID":"ba4af621-bf34-49a3-8870-ecbd873d50b2","Type":"ContainerDied","Data":"992c7944f90bbf21b36b1a3e10000e39aaefbe99306e851d8713710ee28b795d"}
Jul 31 13:16:12.641118 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.641020    2586 scope.go:117] "RemoveContainer" containerID="b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38"
Jul 31 13:16:12.652505 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.652484    2586 scope.go:117] "RemoveContainer" containerID="bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5"
Jul 31 13:16:12.661834 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.661817    2586 scope.go:117] "RemoveContainer" containerID="b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38"
Jul 31 13:16:12.662108 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:16:12.662088    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38\": container with ID starting with b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38 not found: ID does not exist" containerID="b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38"
Jul 31 13:16:12.662179 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.662115    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38"} err="failed to get container status \"b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38\": rpc error: code = NotFound desc = could not find container \"b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38\": container with ID starting with b6552c3d6023907631cfa8a94141838a5f21cc098fef12b0648e546fcea2da38 not found: ID does not exist"
Jul 31 13:16:12.662179 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.662132    2586 scope.go:117] "RemoveContainer" containerID="bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5"
Jul 31 13:16:12.662419 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:16:12.662399    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5\": container with ID starting with bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5 not found: ID does not exist" containerID="bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5"
Jul 31 13:16:12.662468 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.662426    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5"} err="failed to get container status \"bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5\": rpc error: code = NotFound desc = could not find container \"bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5\": container with ID starting with bad2a93ae478145ca1eb098a84a094396e4cad8f01ec13828179f3544a4707a5 not found: ID does not exist"
Jul 31 13:16:12.763009 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.762982    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-envoy\") pod \"ba4af621-bf34-49a3-8870-ecbd873d50b2\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") "
Jul 31 13:16:12.763175 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763051    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9rkv\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-kube-api-access-d9rkv\") pod \"ba4af621-bf34-49a3-8870-ecbd873d50b2\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") "
Jul 31 13:16:12.763175 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763077    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-podinfo\") pod \"ba4af621-bf34-49a3-8870-ecbd873d50b2\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") "
Jul 31 13:16:12.763175 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763100    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-token\") pod \"ba4af621-bf34-49a3-8870-ecbd873d50b2\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") "
Jul 31 13:16:12.763175 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763125    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4af621-bf34-49a3-8870-ecbd873d50b2-istiod-ca-cert\") pod \"ba4af621-bf34-49a3-8870-ecbd873d50b2\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") "
Jul 31 13:16:12.763175 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763160    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-socket\") pod \"ba4af621-bf34-49a3-8870-ecbd873d50b2\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") "
Jul 31 13:16:12.763848 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763183    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-data\") pod \"ba4af621-bf34-49a3-8870-ecbd873d50b2\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") "
Jul 31 13:16:12.763848 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763199    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-certs\") pod \"ba4af621-bf34-49a3-8870-ecbd873d50b2\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") "
Jul 31 13:16:12.763848 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763228    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-credential-socket\") pod \"ba4af621-bf34-49a3-8870-ecbd873d50b2\" (UID: \"ba4af621-bf34-49a3-8870-ecbd873d50b2\") "
Jul 31 13:16:12.763848 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763591    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4af621-bf34-49a3-8870-ecbd873d50b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba4af621-bf34-49a3-8870-ecbd873d50b2" (UID: "ba4af621-bf34-49a3-8870-ecbd873d50b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:16:12.763848 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763611    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba4af621-bf34-49a3-8870-ecbd873d50b2" (UID: "ba4af621-bf34-49a3-8870-ecbd873d50b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:16:12.763848 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763721    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba4af621-bf34-49a3-8870-ecbd873d50b2" (UID: "ba4af621-bf34-49a3-8870-ecbd873d50b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:16:12.763848 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763765    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba4af621-bf34-49a3-8870-ecbd873d50b2" (UID: "ba4af621-bf34-49a3-8870-ecbd873d50b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:16:12.764230 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.763892    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba4af621-bf34-49a3-8870-ecbd873d50b2" (UID: "ba4af621-bf34-49a3-8870-ecbd873d50b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:16:12.765329 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.765302    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba4af621-bf34-49a3-8870-ecbd873d50b2" (UID: "ba4af621-bf34-49a3-8870-ecbd873d50b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 13:16:12.765422 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.765394    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba4af621-bf34-49a3-8870-ecbd873d50b2" (UID: "ba4af621-bf34-49a3-8870-ecbd873d50b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:16:12.765605 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.765585    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-kube-api-access-d9rkv" (OuterVolumeSpecName: "kube-api-access-d9rkv") pod "ba4af621-bf34-49a3-8870-ecbd873d50b2" (UID: "ba4af621-bf34-49a3-8870-ecbd873d50b2"). InnerVolumeSpecName "kube-api-access-d9rkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:16:12.765679 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.765593    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba4af621-bf34-49a3-8870-ecbd873d50b2" (UID: "ba4af621-bf34-49a3-8870-ecbd873d50b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 13:16:12.864815 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.864790    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:12.864815 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.864815    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:12.865005 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.864827    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:12.865005 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.864840    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:12.865005 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.864851    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:12.865005 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.864864    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9rkv\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-kube-api-access-d9rkv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:12.865005 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.864875    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:12.865005 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.864886    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4af621-bf34-49a3-8870-ecbd873d50b2-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:12.865005 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.864896    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4af621-bf34-49a3-8870-ecbd873d50b2-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:12.961870 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.961842    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"]
Jul 31 13:16:12.966025 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:12.966002    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-96b3e0-openshift-default-594d9c6588-sbx2k"]
Jul 31 13:16:13.222830 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.222741    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"]
Jul 31 13:16:13.222990 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.222961    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml" podUID="ae7d0d5d-d6d4-4190-af32-33163ce28fc4" containerName="maas-api" containerID="cri-o://1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac" gracePeriod=30
Jul 31 13:16:13.288813 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.288781    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"]
Jul 31 13:16:13.289043 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.289018    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j" podUID="29b92a90-2104-4ba9-b336-54fb93b3f16a" containerName="payload-pre-processing" containerID="cri-o://c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d" gracePeriod=30
Jul 31 13:16:13.535410 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.535388    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:16:13.563629 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.563606    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:16:13.646959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.646919    2586 generic.go:356] "Generic (PLEG): container finished" podID="ae7d0d5d-d6d4-4190-af32-33163ce28fc4" containerID="1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac" exitCode=0
Jul 31 13:16:13.646959 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.646962    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"
Jul 31 13:16:13.647473 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.646999    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml" event={"ID":"ae7d0d5d-d6d4-4190-af32-33163ce28fc4","Type":"ContainerDied","Data":"1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac"}
Jul 31 13:16:13.647473 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.647042    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml" event={"ID":"ae7d0d5d-d6d4-4190-af32-33163ce28fc4","Type":"ContainerDied","Data":"5d7ad36ccdd9a17a5df6a69aaea652857751a7076ae8ccb47a2f06c07213f6fd"}
Jul 31 13:16:13.647473 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.647072    2586 scope.go:117] "RemoveContainer" containerID="1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac"
Jul 31 13:16:13.648421 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.648400    2586 generic.go:356] "Generic (PLEG): container finished" podID="29b92a90-2104-4ba9-b336-54fb93b3f16a" containerID="c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d" exitCode=0
Jul 31 13:16:13.648526 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.648444    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"
Jul 31 13:16:13.648526 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.648482    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j" event={"ID":"29b92a90-2104-4ba9-b336-54fb93b3f16a","Type":"ContainerDied","Data":"c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d"}
Jul 31 13:16:13.648526 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.648522    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j" event={"ID":"29b92a90-2104-4ba9-b336-54fb93b3f16a","Type":"ContainerDied","Data":"ed1cd38b5ba29e21972c447fd09820919e76ea17919ee4bf5d02cedebfeb5d3d"}
Jul 31 13:16:13.656663 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.656630    2586 scope.go:117] "RemoveContainer" containerID="1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac"
Jul 31 13:16:13.656935 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:16:13.656918    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac\": container with ID starting with 1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac not found: ID does not exist" containerID="1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac"
Jul 31 13:16:13.657008 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.656948    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac"} err="failed to get container status \"1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac\": rpc error: code = NotFound desc = could not find container \"1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac\": container with ID starting with 1ae03e7a1a79d618aedbc9e85886f848beb5024b542b0535c5a3df0e174d18ac not found: ID does not exist"
Jul 31 13:16:13.657008 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.656987    2586 scope.go:117] "RemoveContainer" containerID="c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d"
Jul 31 13:16:13.664386 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.664371    2586 scope.go:117] "RemoveContainer" containerID="c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d"
Jul 31 13:16:13.664657 ip-10-0-136-35 kubenswrapper[2586]: E0731 13:16:13.664630    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d\": container with ID starting with c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d not found: ID does not exist" containerID="c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d"
Jul 31 13:16:13.664727 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.664666    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d"} err="failed to get container status \"c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d\": rpc error: code = NotFound desc = could not find container \"c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d\": container with ID starting with c000cfdbbee6476688402d1f03a1d39268a5bdfe033522dc7b0740cec060923d not found: ID does not exist"
Jul 31 13:16:13.671930 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.671912    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-openshift-service-ca\") pod \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") "
Jul 31 13:16:13.671996 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.671949    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2895\" (UniqueName: \"kubernetes.io/projected/29b92a90-2104-4ba9-b336-54fb93b3f16a-kube-api-access-b2895\") pod \"29b92a90-2104-4ba9-b336-54fb93b3f16a\" (UID: \"29b92a90-2104-4ba9-b336-54fb93b3f16a\") "
Jul 31 13:16:13.671996 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.671971    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-maas-api-tls\") pod \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") "
Jul 31 13:16:13.672079 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.672012    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b92a90-2104-4ba9-b336-54fb93b3f16a-plugins-config-volume\") pod \"29b92a90-2104-4ba9-b336-54fb93b3f16a\" (UID: \"29b92a90-2104-4ba9-b336-54fb93b3f16a\") "
Jul 31 13:16:13.672079 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.672049    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nst2n\" (UniqueName: \"kubernetes.io/projected/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-kube-api-access-nst2n\") pod \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\" (UID: \"ae7d0d5d-d6d4-4190-af32-33163ce28fc4\") "
Jul 31 13:16:13.672280 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.672237    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae7d0d5d-d6d4-4190-af32-33163ce28fc4" (UID: "ae7d0d5d-d6d4-4190-af32-33163ce28fc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:16:13.672420 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.672397    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b92a90-2104-4ba9-b336-54fb93b3f16a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29b92a90-2104-4ba9-b336-54fb93b3f16a" (UID: "29b92a90-2104-4ba9-b336-54fb93b3f16a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 13:16:13.672501 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.672425    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:13.673950 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.673930    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b92a90-2104-4ba9-b336-54fb93b3f16a-kube-api-access-b2895" (OuterVolumeSpecName: "kube-api-access-b2895") pod "29b92a90-2104-4ba9-b336-54fb93b3f16a" (UID: "29b92a90-2104-4ba9-b336-54fb93b3f16a"). InnerVolumeSpecName "kube-api-access-b2895". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:16:13.673950 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.673940    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae7d0d5d-d6d4-4190-af32-33163ce28fc4" (UID: "ae7d0d5d-d6d4-4190-af32-33163ce28fc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 13:16:13.674064 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.673957    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-kube-api-access-nst2n" (OuterVolumeSpecName: "kube-api-access-nst2n") pod "ae7d0d5d-d6d4-4190-af32-33163ce28fc4" (UID: "ae7d0d5d-d6d4-4190-af32-33163ce28fc4"). InnerVolumeSpecName "kube-api-access-nst2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 13:16:13.773252 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.773172    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2895\" (UniqueName: \"kubernetes.io/projected/29b92a90-2104-4ba9-b336-54fb93b3f16a-kube-api-access-b2895\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:13.773252 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.773201    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:13.773252 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.773215    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b92a90-2104-4ba9-b336-54fb93b3f16a-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:13.773252 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.773227    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nst2n\" (UniqueName: \"kubernetes.io/projected/ae7d0d5d-d6d4-4190-af32-33163ce28fc4-kube-api-access-nst2n\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\""
Jul 31 13:16:13.967491 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.967460    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"]
Jul 31 13:16:13.973385 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.973354    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e55b67-bb66bc565-j6qml"]
Jul 31 13:16:13.983705 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.983682    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"]
Jul 31 13:16:13.990064 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:13.990042    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e55b67-75b7d76db4-wjf8j"]
Jul 31 13:16:14.485338 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:14.485307    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b92a90-2104-4ba9-b336-54fb93b3f16a" path="/var/lib/kubelet/pods/29b92a90-2104-4ba9-b336-54fb93b3f16a/volumes"
Jul 31 13:16:14.485687 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:14.485674    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae7d0d5d-d6d4-4190-af32-33163ce28fc4" path="/var/lib/kubelet/pods/ae7d0d5d-d6d4-4190-af32-33163ce28fc4/volumes"
Jul 31 13:16:14.486090 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:14.486076    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" path="/var/lib/kubelet/pods/ba4af621-bf34-49a3-8870-ecbd873d50b2/volumes"
Jul 31 13:16:20.726354 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:20.726314    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bcd57d65f-pqfnn_b9bcd47f-94e1-45dd-81e1-36ce04c9c303/authorino/0.log"
Jul 31 13:16:26.191113 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:26.191080    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fb9rn_313c0715-e29e-4020-a8bf-f53709170b4b/manager/0.log"
Jul 31 13:16:26.445731 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:26.445650    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c67bf5868-njj7d_0e72d399-87d9-4a32-9c99-5e44b4616a22/manager/0.log"
Jul 31 13:16:26.561992 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:26.561960    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9sgdg_f8149d5f-9913-492e-a13b-10e4b66e4145/server/0.log"
Jul 31 13:16:26.686965 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:26.686936    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v6mcf_bdc9f096-64da-49ad-928f-c3abdcc63294/manager/0.log"
Jul 31 13:16:26.805740 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:26.805657    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996b4548f-8pcwp_28d114e9-c01c-4f60-8c23-b04aafdd2d38/copy-manifests/0.log"
Jul 31 13:16:26.823697 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:26.823664    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996b4548f-8pcwp_28d114e9-c01c-4f60-8c23-b04aafdd2d38/manager/0.log"
Jul 31 13:16:27.423559 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:27.423508    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b9d74c676-4zjcp_3e162864-1ba6-4afe-97ee-b6ec98606d1c/maas-api/0.log"
Jul 31 13:16:27.534761 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:27.534726    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29758365-jhkbh_560ddcb9-4b7c-4459-a03c-d3e28cad40f6/cleanup/0.log"
Jul 31 13:16:27.864328 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:27.864247    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-7rdnc_1b436784-bb49-45ce-a75f-d33343637676/postgres/0.log"
Jul 31 13:16:28.645435 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:28.645399    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bcd57d65f-pqfnn_b9bcd47f-94e1-45dd-81e1-36ce04c9c303/authorino/0.log"
Jul 31 13:16:28.763609 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:28.763579    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-vh8kb_5e7c23f0-dacb-4ef0-b03b-f887654f123f/manager/0.log"
Jul 31 13:16:28.883366 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:28.883163    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-xpxx2_28d9fb8b-f4c9-4bc7-8824-a5f26b716988/manager/0.log"
Jul 31 13:16:28.992431 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:28.992406    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-q989n_558fd764-0823-4c71-b5c1-0a756172ff45/kuadrant-console-plugin/0.log"
Jul 31 13:16:29.344676 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:29.344584    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-7cbm7_ca0e3552-5f2d-41ba-9ab9-d13684b5f047/manager/0.log"
Jul 31 13:16:30.036449 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:30.036413    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84cd848656-kxhxj_de430a35-bac6-4020-a36c-8395a5c6e5d5/kube-auth-proxy/0.log"
Jul 31 13:16:30.154016 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:30.153989    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d2l22_4c4e5b62-ec42-4c74-a626-65a8f9f2260e/istio-proxy/1.log"
Jul 31 13:16:30.293003 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:30.292919    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2rx7z_5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc/payload-pre-processing/0.log"
Jul 31 13:16:30.410656 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:30.410625    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pdrrk_d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2/payload-processing/0.log"
Jul 31 13:16:30.883382 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:30.883350    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5gtqg_a2c1ecf6-e394-4a9f-a88e-0b0487221521/main/0.log"
Jul 31 13:16:31.133348 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:31.133321    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-bln7l_880003dc-67b7-41a8-9276-24cdc2822714/main/0.log"
Jul 31 13:16:31.360200 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:31.360172    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-fb6jd_ea7b93cf-7a1d-4b45-867a-2992bcd2fa51/main/0.log"
Jul 31 13:16:36.583138 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583064    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c6r5k/must-gather-5wqwl"]
Jul 31 13:16:36.583514 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583471    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7d0d5d-d6d4-4190-af32-33163ce28fc4" containerName="maas-api"
Jul 31 13:16:36.583514 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583485    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7d0d5d-d6d4-4190-af32-33163ce28fc4" containerName="maas-api"
Jul 31 13:16:36.583514 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583495    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b92a90-2104-4ba9-b336-54fb93b3f16a" containerName="payload-pre-processing"
Jul 31 13:16:36.583514 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583500    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b92a90-2104-4ba9-b336-54fb93b3f16a" containerName="payload-pre-processing"
Jul 31 13:16:36.583514 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583512    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931d1041-1a96-407f-a1ff-dac0e9a294bf" containerName="payload-pre-processing"
Jul 31 13:16:36.583514 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583518    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="931d1041-1a96-407f-a1ff-dac0e9a294bf" containerName="payload-pre-processing"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583525    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558ec461-aa59-4217-a669-309c134b7664" containerName="payload-processing"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583531    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="558ec461-aa59-4217-a669-309c134b7664" containerName="payload-processing"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583536    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01fd4400-bb38-4f3a-9eb7-4119889c64fa" containerName="maas-api"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583542    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="01fd4400-bb38-4f3a-9eb7-4119889c64fa" containerName="maas-api"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583550    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be02cd63-0cc4-40b3-8f5f-6e743988f358" containerName="revoke-keys"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583555    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="be02cd63-0cc4-40b3-8f5f-6e743988f358" containerName="revoke-keys"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583566    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583571    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583577    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583582    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583638    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="931d1041-1a96-407f-a1ff-dac0e9a294bf" containerName="payload-pre-processing"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583647    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="be02cd63-0cc4-40b3-8f5f-6e743988f358" containerName="revoke-keys"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583655    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="558ec461-aa59-4217-a669-309c134b7664" containerName="payload-processing"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583660    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583665    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b92a90-2104-4ba9-b336-54fb93b3f16a" containerName="payload-pre-processing"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583672    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba4af621-bf34-49a3-8870-ecbd873d50b2" containerName="istio-proxy"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583677    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="01fd4400-bb38-4f3a-9eb7-4119889c64fa" containerName="maas-api"
Jul 31 13:16:36.583713 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.583682    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7d0d5d-d6d4-4190-af32-33163ce28fc4" containerName="maas-api"
Jul 31 13:16:36.586762 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.586745    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c6r5k/must-gather-5wqwl"
Jul 31 13:16:36.589142 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.589117    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c6r5k\"/\"openshift-service-ca.crt\""
Jul 31 13:16:36.589277 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.589144    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c6r5k\"/\"kube-root-ca.crt\""
Jul 31 13:16:36.589277 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.589233    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c6r5k\"/\"default-dockercfg-bpt22\""
Jul 31 13:16:36.596144 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.596124    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c6r5k/must-gather-5wqwl"]
Jul 31 13:16:36.663090 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.663061    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69tw\" (UniqueName: \"kubernetes.io/projected/f952dc36-9228-4060-b21c-569f97d839a4-kube-api-access-n69tw\") pod \"must-gather-5wqwl\" (UID: \"f952dc36-9228-4060-b21c-569f97d839a4\") " pod="openshift-must-gather-c6r5k/must-gather-5wqwl"
Jul 31 13:16:36.663293 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.663108    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f952dc36-9228-4060-b21c-569f97d839a4-must-gather-output\") pod \"must-gather-5wqwl\" (UID: \"f952dc36-9228-4060-b21c-569f97d839a4\") " pod="openshift-must-gather-c6r5k/must-gather-5wqwl"
Jul 31 13:16:36.763868 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.763837    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69tw\" (UniqueName: \"kubernetes.io/projected/f952dc36-9228-4060-b21c-569f97d839a4-kube-api-access-n69tw\") pod \"must-gather-5wqwl\" (UID: \"f952dc36-9228-4060-b21c-569f97d839a4\") " pod="openshift-must-gather-c6r5k/must-gather-5wqwl"
Jul 31 13:16:36.764049 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.763897    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f952dc36-9228-4060-b21c-569f97d839a4-must-gather-output\") pod \"must-gather-5wqwl\" (UID: \"f952dc36-9228-4060-b21c-569f97d839a4\") " pod="openshift-must-gather-c6r5k/must-gather-5wqwl"
Jul 31 13:16:36.764241 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.764225    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f952dc36-9228-4060-b21c-569f97d839a4-must-gather-output\") pod \"must-gather-5wqwl\" (UID: \"f952dc36-9228-4060-b21c-569f97d839a4\") " pod="openshift-must-gather-c6r5k/must-gather-5wqwl"
Jul 31 13:16:36.771377 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.771349    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69tw\" (UniqueName: \"kubernetes.io/projected/f952dc36-9228-4060-b21c-569f97d839a4-kube-api-access-n69tw\") pod \"must-gather-5wqwl\" (UID: \"f952dc36-9228-4060-b21c-569f97d839a4\") " pod="openshift-must-gather-c6r5k/must-gather-5wqwl"
Jul 31 13:16:36.896766 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:36.896695    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c6r5k/must-gather-5wqwl"
Jul 31 13:16:37.027202 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:37.027175    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c6r5k/must-gather-5wqwl"]
Jul 31 13:16:37.029132 ip-10-0-136-35 kubenswrapper[2586]: W0731 13:16:37.029100    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf952dc36_9228_4060_b21c_569f97d839a4.slice/crio-b07a5a522fa1bd8e2c896ed43a395de9762e69f0b878b75326329812aeac8adb WatchSource:0}: Error finding container b07a5a522fa1bd8e2c896ed43a395de9762e69f0b878b75326329812aeac8adb: Status 404 returned error can't find the container with id b07a5a522fa1bd8e2c896ed43a395de9762e69f0b878b75326329812aeac8adb
Jul 31 13:16:37.030841 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:37.030825    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 13:16:37.767372 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:37.767337    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c6r5k/must-gather-5wqwl" event={"ID":"f952dc36-9228-4060-b21c-569f97d839a4","Type":"ContainerStarted","Data":"b07a5a522fa1bd8e2c896ed43a395de9762e69f0b878b75326329812aeac8adb"}
Jul 31 13:16:38.776377 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:38.776328    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c6r5k/must-gather-5wqwl" event={"ID":"f952dc36-9228-4060-b21c-569f97d839a4","Type":"ContainerStarted","Data":"5f4c3348543807cfa98ea58f6b612e5cd0dd76b6d8fd96f93d1f7426bbe0824e"}
Jul 31 13:16:38.776888 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:38.776386    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c6r5k/must-gather-5wqwl" event={"ID":"f952dc36-9228-4060-b21c-569f97d839a4","Type":"ContainerStarted","Data":"0bd511dd55df8cb71d15bc112a67fe8f5f22673544c8c4b2afceb1ad5accde7b"}
Jul 31 13:16:38.793147 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:38.793097    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c6r5k/must-gather-5wqwl" podStartSLOduration=1.943697807 podStartE2EDuration="2.793081556s" totalImagesPullingTime="849.383749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 13:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:16:38.791062912 +0000 UTC m=+3686.778284453" watchObservedRunningTime="2026-07-31 13:16:38.793081556 +0000 UTC m=+3686.780303096"
Jul 31 13:16:39.393041 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:39.393013    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pwtgz_e0a25e8a-583a-4194-a8df-ff565cac7a43/global-pull-secret-syncer/0.log"
Jul 31 13:16:39.543119 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:39.543084    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jsrdj_27c4f51d-6014-4c7c-8562-034f42cbf93e/konnectivity-agent/0.log"
Jul 31 13:16:39.564487 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:39.564458    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-35.ec2.internal_d64eb269ad67e5f98e8a744fc0f2cae5/haproxy/0.log"
Jul 31 13:16:44.750285 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:44.749706    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bcd57d65f-pqfnn_b9bcd47f-94e1-45dd-81e1-36ce04c9c303/authorino/0.log"
Jul 31 13:16:44.795057 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:44.795006    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-vh8kb_5e7c23f0-dacb-4ef0-b03b-f887654f123f/manager/0.log"
Jul 31 13:16:44.825295 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:44.825003    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-xpxx2_28d9fb8b-f4c9-4bc7-8824-a5f26b716988/manager/0.log"
Jul 31 13:16:44.853587 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:44.853560    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-q989n_558fd764-0823-4c71-b5c1-0a756172ff45/kuadrant-console-plugin/0.log"
Jul 31 13:16:45.093433 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:45.090978    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-7cbm7_ca0e3552-5f2d-41ba-9ab9-d13684b5f047/manager/0.log"
Jul 31 13:16:46.817208 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:46.817172    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c86rz_05ee5666-45cf-495d-8c43-5f4435d6f7d9/node-exporter/0.log"
Jul 31 13:16:46.837255 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:46.837228    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c86rz_05ee5666-45cf-495d-8c43-5f4435d6f7d9/kube-rbac-proxy/0.log"
Jul 31 13:16:46.864834 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:46.864754    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c86rz_05ee5666-45cf-495d-8c43-5f4435d6f7d9/init-textfile/0.log"
Jul 31 13:16:49.328496 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.328464    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"]
Jul 31 13:16:49.337058 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.337033    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.340970 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.340948    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"]
Jul 31 13:16:49.401418 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.401382    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-sys\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.401746 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.401720    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-podres\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.401882 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.401865    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fn7\" (UniqueName: \"kubernetes.io/projected/af6a18c6-380e-41fc-934a-ca8453cb5399-kube-api-access-v4fn7\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.402007 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.401989    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-proc\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.402230 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.402209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-lib-modules\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.503635 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.503598    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-sys\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.504002 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.503904    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-sys\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.504230 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.504156    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-podres\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.504230 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.504200    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fn7\" (UniqueName: \"kubernetes.io/projected/af6a18c6-380e-41fc-934a-ca8453cb5399-kube-api-access-v4fn7\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.504397 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.504234    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-proc\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.504397 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.504343    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-lib-modules\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.504519 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.504498    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-lib-modules\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.504630 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.504613    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-podres\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.504760 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.504745    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af6a18c6-380e-41fc-934a-ca8453cb5399-proc\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.513258 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.513234    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fn7\" (UniqueName: \"kubernetes.io/projected/af6a18c6-380e-41fc-934a-ca8453cb5399-kube-api-access-v4fn7\") pod \"perf-node-gather-daemonset-k9gkg\" (UID: \"af6a18c6-380e-41fc-934a-ca8453cb5399\") " pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.655276 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.655229    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:49.858586 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.857395    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"]
Jul 31 13:16:49.871977 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:49.871756    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg" event={"ID":"af6a18c6-380e-41fc-934a-ca8453cb5399","Type":"ContainerStarted","Data":"758fe8029920cc3c13a56e954844960348e6dfa205cf11a908150b05010c71a9"}
Jul 31 13:16:50.879848 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:50.879802    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg" event={"ID":"af6a18c6-380e-41fc-934a-ca8453cb5399","Type":"ContainerStarted","Data":"a8ee8dcfb3da1b02da78cf948403af20e183e42a0bb3919b2e6ec23428774922"}
Jul 31 13:16:50.880779 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:50.880752    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:50.897314 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:50.897244    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg" podStartSLOduration=1.89722203 podStartE2EDuration="1.89722203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 13:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 13:16:50.895206614 +0000 UTC m=+3698.882428155" watchObservedRunningTime="2026-07-31 13:16:50.89722203 +0000 UTC m=+3698.884443571"
Jul 31 13:16:51.836220 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:51.836177    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9d7l8_6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa/dns/0.log"
Jul 31 13:16:51.854711 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:51.854683    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9d7l8_6b60bfd3-dea9-4b4a-aa2b-637e4c04ceaa/kube-rbac-proxy/0.log"
Jul 31 13:16:51.972079 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:51.972052    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-plkjn_bfd73de9-7b12-4de5-ab9b-875c7bf9d62c/dns-node-resolver/0.log"
Jul 31 13:16:52.501168 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:52.501124    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-869447d669-jp8c2_de572c0f-8fb3-4369-9abd-ecf748dfd05b/registry/0.log"
Jul 31 13:16:52.568712 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:52.568679    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jl8fm_70b1fc55-23f2-41da-981d-51d4f737c4e5/node-ca/0.log"
Jul 31 13:16:53.702127 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:53.702100    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84cd848656-kxhxj_de430a35-bac6-4020-a36c-8395a5c6e5d5/kube-auth-proxy/0.log"
Jul 31 13:16:53.731724 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:53.731692    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d2l22_4c4e5b62-ec42-4c74-a626-65a8f9f2260e/istio-proxy/0.log"
Jul 31 13:16:53.745566 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:53.745540    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d2l22_4c4e5b62-ec42-4c74-a626-65a8f9f2260e/istio-proxy/1.log"
Jul 31 13:16:53.784379 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:53.784351    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2rx7z_5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc/payload-pre-processing/0.log"
Jul 31 13:16:53.831874 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:53.831847    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pdrrk_d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2/payload-processing/0.log"
Jul 31 13:16:54.655855 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:54.655823    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2qljl_7b7772db-0980-4ddb-ade7-cf9c5ec44a40/serve-healthcheck-canary/0.log"
Jul 31 13:16:55.222007 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:55.221970    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-74qwb_6b1b9f55-0c73-4d10-bb72-96adf76f8543/kube-rbac-proxy/0.log"
Jul 31 13:16:55.242239 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:55.242199    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-74qwb_6b1b9f55-0c73-4d10-bb72-96adf76f8543/exporter/0.log"
Jul 31 13:16:55.261692 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:55.261668    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-74qwb_6b1b9f55-0c73-4d10-bb72-96adf76f8543/extractor/0.log"
Jul 31 13:16:57.383926 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:57.383901    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fb9rn_313c0715-e29e-4020-a8bf-f53709170b4b/manager/0.log"
Jul 31 13:16:57.904729 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:57.904702    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c6r5k/perf-node-gather-daemonset-k9gkg"
Jul 31 13:16:58.145880 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:58.145837    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c67bf5868-njj7d_0e72d399-87d9-4a32-9c99-5e44b4616a22/manager/0.log"
Jul 31 13:16:58.167638 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:58.167559    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9sgdg_f8149d5f-9913-492e-a13b-10e4b66e4145/server/0.log"
Jul 31 13:16:58.319385 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:58.319351    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v6mcf_bdc9f096-64da-49ad-928f-c3abdcc63294/manager/0.log"
Jul 31 13:16:58.503106 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:58.502734    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996b4548f-8pcwp_28d114e9-c01c-4f60-8c23-b04aafdd2d38/manager/0.log"
Jul 31 13:16:58.522477 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:58.522454    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996b4548f-8pcwp_28d114e9-c01c-4f60-8c23-b04aafdd2d38/copy-manifests/0.log"
Jul 31 13:16:59.950159 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:59.950131    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84cd848656-kxhxj_de430a35-bac6-4020-a36c-8395a5c6e5d5/kube-auth-proxy/0.log"
Jul 31 13:16:59.986377 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:59.986342    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d2l22_4c4e5b62-ec42-4c74-a626-65a8f9f2260e/istio-proxy/0.log"
Jul 31 13:16:59.995224 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:16:59.995152    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d2l22_4c4e5b62-ec42-4c74-a626-65a8f9f2260e/istio-proxy/1.log"
Jul 31 13:17:00.025711 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:00.025674    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2rx7z_5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc/payload-pre-processing/0.log"
Jul 31 13:17:00.053016 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:00.052985    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pdrrk_d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2/payload-processing/0.log"
Jul 31 13:17:01.750893 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:01.750863    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84cd848656-kxhxj_de430a35-bac6-4020-a36c-8395a5c6e5d5/kube-auth-proxy/0.log"
Jul 31 13:17:01.773798 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:01.773767    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d2l22_4c4e5b62-ec42-4c74-a626-65a8f9f2260e/istio-proxy/0.log"
Jul 31 13:17:01.782227 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:01.782203    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d2l22_4c4e5b62-ec42-4c74-a626-65a8f9f2260e/istio-proxy/1.log"
Jul 31 13:17:01.823782 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:01.823756    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2rx7z_5ff2dd07-a0e6-4073-91d9-12e7e2bf72cc/payload-pre-processing/0.log"
Jul 31 13:17:01.852951 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:01.852919    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pdrrk_d2a4ef6f-d103-4ce3-b14c-a5020a0c2ae2/payload-processing/0.log"
Jul 31 13:17:06.470232 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.470208    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8sxf_b1074883-7ab6-46a5-a22c-07e17d691672/kube-multus-additional-cni-plugins/0.log"
Jul 31 13:17:06.487887 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.487861    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8sxf_b1074883-7ab6-46a5-a22c-07e17d691672/egress-router-binary-copy/0.log"
Jul 31 13:17:06.506034 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.506011    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8sxf_b1074883-7ab6-46a5-a22c-07e17d691672/cni-plugins/0.log"
Jul 31 13:17:06.523991 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.523968    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8sxf_b1074883-7ab6-46a5-a22c-07e17d691672/bond-cni-plugin/0.log"
Jul 31 13:17:06.542661 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.542639    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8sxf_b1074883-7ab6-46a5-a22c-07e17d691672/routeoverride-cni/0.log"
Jul 31 13:17:06.561858 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.561834    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8sxf_b1074883-7ab6-46a5-a22c-07e17d691672/whereabouts-cni-bincopy/0.log"
Jul 31 13:17:06.586006 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.585977    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8sxf_b1074883-7ab6-46a5-a22c-07e17d691672/whereabouts-cni/0.log"
Jul 31 13:17:06.779031 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.778960    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lhqd7_eb11a5be-943c-4a5c-8ce3-06b1fab1c840/kube-multus/0.log"
Jul 31 13:17:06.875176 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.875147    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cf94v_7e2e533b-8f29-4a31-a152-c3f07d8d95f1/network-metrics-daemon/0.log"
Jul 31 13:17:06.888707 ip-10-0-136-35 kubenswrapper[2586]: I0731 13:17:06.888674    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cf94v_7e2e533b-8f29-4a31-a152-c3f07d8d95f1/kube-rbac-proxy/0.log"