Sep 10 23:01:40.180904 ip-10-0-135-96 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 23:01:40.584364 ip-10-0-135-96 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:01:40.584364 ip-10-0-135-96 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 23:01:40.584364 ip-10-0-135-96 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:01:40.584364 ip-10-0-135-96 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 23:01:40.584364 ip-10-0-135-96 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:01:40.586273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.586182    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 23:01:40.590015 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.589998    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:40.590015 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590014    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590017    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590020    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590023    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590026    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590029    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590031    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590034    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590037    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590039    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590043    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590045    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590048    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590052    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590055    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590058    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590060    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590063    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590065    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590068    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:40.590079 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590071    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590073    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590075    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590078    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590081    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590084    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590087    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590090    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590092    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590095    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590098    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590101    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590103    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590106    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590110    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590114    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590117    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590119    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590122    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:40.590559 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590125    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590128    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590131    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590133    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590135    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590138    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590140    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590144    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590146    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590149    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590151    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590154    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590156    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590159    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590161    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590164    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590167    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590169    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590173    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590175    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:40.591152 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590178    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590180    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590183    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590185    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590188    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590191    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590193    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590197    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590201    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590204    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590206    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590209    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590212    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590214    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590217    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590220    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590222    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590226    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590229    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590231    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:40.591667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590234    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590237    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590240    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590242    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590245    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590247    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590250    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590253    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590256    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590258    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590261    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590263    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590266    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590269    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590273    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590276    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590278    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590280    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.590283    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:40.592137 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591090    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591097    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591099    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591102    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591105    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591108    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591110    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591113    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591116    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591118    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591121    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591124    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591126    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591129    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591132    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591134    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591137    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591140    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591142    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591145    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:40.592614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591148    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591151    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591154    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591156    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591159    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591161    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591164    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591167    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591170    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591172    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591175    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591177    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591180    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591182    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591185    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591187    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591190    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591193    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591196    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:40.593105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591198    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591201    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591203    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591205    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591208    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591211    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591213    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591215    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591219    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591222    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591224    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591227    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591229    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591233    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591235    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591238    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591240    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591245    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591248    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591251    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:40.593580 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591253    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591256    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591258    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591261    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591263    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591265    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591268    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591270    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591273    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591275    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591278    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591280    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591283    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591287    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591291    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591294    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591297    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591301    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591304    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:40.594077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591307    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591310    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591314    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591317    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591320    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591322    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591325    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591328    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591330    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591333    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591335    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591338    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591340    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591343    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591345    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591348    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591350    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591353    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591355    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591358    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:40.594564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591360    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591816    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591822    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591825    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591828    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591831    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591834    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591836    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591839    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591842    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591844    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591847    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591849    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591852    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591856    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591859    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591861    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591864    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591866    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591869    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:40.595048 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591872    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591874    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591877    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591879    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591882    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591884    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591887    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591889    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591892    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591894    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591898    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591901    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591904    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591908    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591910    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591913    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591916    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591919    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591923    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:40.595643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591925    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591927    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591931    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591933    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591936    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591939    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591942    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591944    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591949    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591951    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591954    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591957    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591959    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591962    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591964    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591966    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591969    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591971    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591973    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591976    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:40.596121 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591978    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591981    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591983    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591985    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591988    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591990    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591993    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591995    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.591997    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592000    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592002    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592004    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592007    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592009    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592011    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592018    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592021    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592024    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592027    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592030    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:40.596615 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592032    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592035    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592038    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592041    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592043    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592046    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592048    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592051    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592053    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592056    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592058    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592061    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592063    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592065    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592068    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592070    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592072    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592075    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592077    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592079    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:40.597105 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592082    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592163    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592204    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592212    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592217    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592221    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592224    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592228    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592232    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592235    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592239    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592242    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592246    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592249    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592253    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592256    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592259    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592262    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592265    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592269    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592272    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592275    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592278    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592281    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 23:01:40.597681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592284    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592287    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592290    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592293    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592296    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592308    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592311    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592314    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592320    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592324    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592327    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592330    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592333    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592336    2580 flags.go:64] FLAG: --enable-server="true"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592339    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592343    2580 flags.go:64] FLAG: --event-burst="100"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592346    2580 flags.go:64] FLAG: --event-qps="50"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592349    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592352    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592355    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592358    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592361    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592364    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592368    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592371    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 23:01:40.598408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592374    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592377    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592380    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592382    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592385    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592388    2580 flags.go:64] FLAG: --feature-gates=""
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592391    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592394    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592397    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592400    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592404    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592407    2580 flags.go:64] FLAG: --help="false"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592410    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592412    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592415    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592419    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592422    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592426    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592428    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592431    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592434    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592436    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592439    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592443    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 23:01:40.599167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592445    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592451    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592454    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592457    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592460    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592462    2580 flags.go:64] FLAG: --lock-file=""
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592474    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592478    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592481    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592491    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592494    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592497    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592500    2580 flags.go:64] FLAG: --logging-format="text"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592503    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592506    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592509    2580 flags.go:64] FLAG: --manifest-url=""
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592511    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592516    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592519    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592523    2580 flags.go:64] FLAG: --max-pods="110"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592526    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592529    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592531    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592536    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592552    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 23:01:40.599781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592555    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592558    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592566    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592569    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592572    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592575    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592578    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592584    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592587    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592592    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592595    2580 flags.go:64] FLAG: --port="10250"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592598    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592601    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b20523598ee91a4b"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592604    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592607    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592610    2580 flags.go:64] FLAG: --register-node="true"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592613    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592617    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592620    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592622    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592625    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592629    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592632    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592635    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592638    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 23:01:40.600669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592641    2580 flags.go:64] FLAG: --runonce="false"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592644    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592647    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592649    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592652    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592656    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592659    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592662    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592665    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592668    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592671    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592674    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592676    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592679    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592682    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592684    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592691    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592694    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592696    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592700    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592703    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592706    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592709    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592712    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592715    2580 flags.go:64] FLAG: --v="2"
Sep 10 23:01:40.601737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592719    2580 flags.go:64] FLAG: --version="false"
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592723    2580 flags.go:64] FLAG: --vmodule=""
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592726    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.592729    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592850    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592853    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592857    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592860    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592862    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592865    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592867    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592870    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592874    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592876    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592879    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592882    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592884    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592887    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592889    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592892    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:40.602768 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592894    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592897    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592899    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592903    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592906    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592908    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592911    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592913    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592916    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592918    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592921    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592925    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592929    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592938    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592942    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592944    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592947    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592949    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592952    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592954    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:40.603256 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592957    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592959    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592962    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592964    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592968    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592972    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592975    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592978    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592981    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592984    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592986    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592989    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592991    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592994    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.592996    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593000    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593002    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593005    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593007    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:40.603901 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593010    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593012    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593015    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593017    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593020    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593023    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593025    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593028    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593031    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593033    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593036    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593038    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593041    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593043    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593045    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593048    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593050    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593054    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593057    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593059    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:40.604758 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593062    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593064    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593067    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593069    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593072    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593074    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593077    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593079    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593084    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593086    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593089    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593091    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593094    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593097    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593099    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593102    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593105    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593108    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593111    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:40.605404 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593115    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593118    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593121    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593125    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.593127    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.593593    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.602425    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.602448    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.603590    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.603606    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603694    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603703    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603708    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603714    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603719    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:40.606108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603724    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603729    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603734    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603738    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603743    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603748    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603752    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603756    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603760    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603764    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603769    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603774    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603778    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603782    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603786    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603790    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603795    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603799    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603804    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603808    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:40.606826 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603812    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603816    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603821    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603826    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603830    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603834    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603838    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603843    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603847    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603851    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603855    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603859    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603865    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603872    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603877    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603882    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603887    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603891    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603895    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:40.607438 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603898    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603903    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603908    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603913    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603917    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603921    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603925    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603929    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603933    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603937    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603942    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603946    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603950    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603954    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603958    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603962    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603967    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603971    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603975    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603979    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:40.608075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603984    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603990    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.603996    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604001    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604005    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604009    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604013    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604017    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604021    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604025    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604029    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604033    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604037    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604041    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604046    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604052    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604056    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604060    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604064    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:40.608665 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604068    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604072    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604077    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604081    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604085    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604089    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604093    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604097    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604101    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604108    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604114    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604119    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604124    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604129    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604133    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:40.609136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604138    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.604146    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604334    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604344    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604349    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604354    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604359    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604363    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604368    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604373    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604377    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604382    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604387    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604392    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604398    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:40.609499 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604402    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604406    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604410    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604414    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604418    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604423    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604427    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604431    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604435    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604439    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604444    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604448    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604453    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604457    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604461    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604474    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604478    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604482    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604486    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604490    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604495    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:40.609944 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604499    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604502    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604506    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604511    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604515    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604522    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604528    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604533    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604555    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604560    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604565    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604571    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604575    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604580    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604584    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604589    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604593    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604598    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604602    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:40.610467 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604607    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604611    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604615    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604620    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604624    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604628    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604632    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604636    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604640    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604644    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604648    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604652    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604656    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604663    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604668    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604673    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604678    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604682    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604686    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:40.610984 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604691    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604695    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604700    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604704    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604708    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604713    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604719    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604724    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604728    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604732    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604736    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604740    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604744    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604748    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604753    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604756    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604760    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604764    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604768    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:40.611456 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604771    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604775    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604780    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604785    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604790    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604794    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604798    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:40.604802    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.604810    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.605010    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.607732    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.609812    2580 server.go:1019] "Starting client certificate rotation"
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.609916    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:01:40.612049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.609951    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:01:40.612737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.612525    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:01:40.613802 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.613780    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:01:40.632828 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.632808    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 10 23:01:40.635664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.635652    2580 log.go:25] "Validated CRI v1 image API"
Sep 10 23:01:40.637458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.637442    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 23:01:40.640017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.640001    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:01:40.641392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.641372    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cb3abdec-1aa5-40a2-af4b-18ed5f889ff7:/dev/nvme0n1p4 d0875cad-0583-41ef-9768-b31450620754:/dev/nvme0n1p3]
Sep 10 23:01:40.641447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.641392    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:30 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 23:01:40.648798 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.648687    2580 manager.go:216] Machine: {Timestamp:2026-09-10 23:01:40.647324606 +0000 UTC m=+0.348957540 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097920 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24455ae8f857842f6c54426a503d54 SystemUUID:ec24455a-e8f8-5784-2f6c-54426a503d54 BootID:bf8b247d-abf8-484c-9812-ec59e10b081e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:30 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:57:15:46:59:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:57:15:46:59:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:ac:90:c3:9e:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 23:01:40.648798 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.648788    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 23:01:40.648947 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.648870    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 23:01:40.651934 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.651907    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 23:01:40.652120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.651938    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 23:01:40.652764 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.652752    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 23:01:40.652818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.652771    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 23:01:40.652818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.652782    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:01:40.653581 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.653569    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:01:40.654377 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.654366    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 23:01:40.657577 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.657563    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:01:40.657721 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.657710    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 23:01:40.657783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.657746    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 23:01:40.657783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.657761    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 23:01:40.657783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.657775    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 23:01:40.657902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.657789    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 10 23:01:40.657902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.657801    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 23:01:40.658161 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.658148    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:01:40.658220 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.658171    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:01:40.660793 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.660773    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 23:01:40.662331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.662316    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 23:01:40.662384 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.662349    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 23:01:40.664097 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664084    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 23:01:40.664140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664106    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 23:01:40.664140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664115    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 23:01:40.664140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664124    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 23:01:40.664140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664136    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 23:01:40.664245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664144    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 23:01:40.664245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664163    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 23:01:40.664245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664171    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 23:01:40.664245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664180    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 23:01:40.664245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664188    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 23:01:40.664245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664200    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 23:01:40.664245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664208    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 23:01:40.664506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664415    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7mk7b"
Sep 10 23:01:40.664929 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664916    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 23:01:40.664960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.664933    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 23:01:40.666505 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.666475    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 23:01:40.666591 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.666514    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 23:01:40.668749 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.668735    2580 server.go:1268] "Started kubelet"
Sep 10 23:01:40.668840 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.668789    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 23:01:40.668897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.668809    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 23:01:40.668897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.668880    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 23:01:40.669701 ip-10-0-135-96 systemd[1]: Started Kubernetes Kubelet.
Sep 10 23:01:40.670789 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.670600    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7mk7b"
Sep 10 23:01:40.670889 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.670876    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 23:01:40.670939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.670913    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 23:01:40.671294 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.671277    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 23:01:40.672763 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.672748    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 23:01:40.676379 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.676365    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 23:01:40.678895 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.678875    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 23:01:40.679017 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.675354    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-96.ec2.internal.18d417f6c8a6e8c3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-96.ec2.internal,UID:ip-10-0-135-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-96.ec2.internal,},FirstTimestamp:2026-09-10 23:01:40.668704963 +0000 UTC m=+0.370337896,LastTimestamp:2026-09-10 23:01:40.668704963 +0000 UTC m=+0.370337896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-96.ec2.internal,}"
Sep 10 23:01:40.679083 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.679060    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:40.680052 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.680035    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 23:01:40.680173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.680156    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 23:01:40.680328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.680312    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 23:01:40.680483 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.680440    2580 factory.go:153] Registering CRI-O factory
Sep 10 23:01:40.680483 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.680476    2580 factory.go:223] Registration of the crio container factory successfully
Sep 10 23:01:40.680681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.680525    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 23:01:40.680681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.680564    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 23:01:40.681624 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.681607    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 23:01:40.681732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.681683    2580 factory.go:55] Registering systemd factory
Sep 10 23:01:40.681732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.681704    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 10 23:01:40.681732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.681722    2580 factory.go:103] Registering Raw factory
Sep 10 23:01:40.681867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.681739    2580 manager.go:1196] Started watching for new ooms in manager
Sep 10 23:01:40.682631 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.682603    2580 manager.go:349] Starting recovery of all containers
Sep 10 23:01:40.684107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.684081    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:40.687436 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.687397    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-96.ec2.internal\" not found" node="ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.688754 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.688690    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 23:01:40.695263 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.694981    2580 manager.go:354] Recovery completed
Sep 10 23:01:40.699792 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.699778    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:40.701797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.701782    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:40.701867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.701810    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:40.701867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.701821    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:40.703150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.703138    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 23:01:40.703207 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.703149    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 23:01:40.703207 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.703171    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:01:40.703207 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.703176    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:40.704039 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.704026    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:40.704094 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.704051    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:40.704094 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.704060    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:40.704291 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.704279    2580 policy_none.go:49] "None policy: Start"
Sep 10 23:01:40.704334 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.704298    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 23:01:40.704334 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.704309    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 23:01:40.705570 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.705559    2580 policy_none.go:47] "Start"
Sep 10 23:01:40.738236 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.738222    2580 manager.go:337] "Starting Device Plugin manager"
Sep 10 23:01:40.738365 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.738261    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 23:01:40.738365 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.738272    2580 server.go:87] "Starting device plugin registration server"
Sep 10 23:01:40.738707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.738690    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 23:01:40.754600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.738703    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 23:01:40.754600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.739008    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 23:01:40.754600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.739138    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 23:01:40.754600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.739146    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 23:01:40.754600 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.739623    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 23:01:40.754600 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.739704    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:40.827065 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.827032    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 23:01:40.827193 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.827093    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 23:01:40.827193 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.827117    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 23:01:40.827258 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.827211    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 23:01:40.829944 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.829922    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:40.839001 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.838934    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:40.840113 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.840098    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:40.840187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.840127    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:40.840187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.840141    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:40.840187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.840162    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.848144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.848129    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.848189 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.848151    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-96.ec2.internal\": node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:40.861796 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.861780    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:40.928254 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.928215    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal"]
Sep 10 23:01:40.928360 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.928304    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:40.929956 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.929941    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:40.930036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.929971    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:40.930036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.929986    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:40.931304 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.931292    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:40.931503 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.931489    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.931536 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.931526    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:40.932214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.932196    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:40.932316 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.932228    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:40.932316 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.932245    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:40.932316 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.932204    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:40.932316 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.932290    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:40.932316 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.932299    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:40.933649 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.933634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.933707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.933670    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:40.934360 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.934346    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:40.934410 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.934375    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:40.934410 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.934389    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:40.959207 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.959182    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-96.ec2.internal\" not found" node="ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.962407 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.962389    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:40.962498 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:40.962482    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-96.ec2.internal\" not found" node="ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.983079 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.983055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3831ac9446a01c59b9333f72ccada707-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal\" (UID: \"3831ac9446a01c59b9333f72ccada707\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.983199 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.983083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3831ac9446a01c59b9333f72ccada707-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal\" (UID: \"3831ac9446a01c59b9333f72ccada707\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:40.983199 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:40.983101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c72a50c000c2c43d429e4d0ccd75117-config\") pod \"kube-apiserver-proxy-ip-10-0-135-96.ec2.internal\" (UID: \"6c72a50c000c2c43d429e4d0ccd75117\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:41.062882 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.062848    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:41.083242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.083212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3831ac9446a01c59b9333f72ccada707-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal\" (UID: \"3831ac9446a01c59b9333f72ccada707\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:41.083242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.083167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3831ac9446a01c59b9333f72ccada707-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal\" (UID: \"3831ac9446a01c59b9333f72ccada707\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:41.083397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.083285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3831ac9446a01c59b9333f72ccada707-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal\" (UID: \"3831ac9446a01c59b9333f72ccada707\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:41.083397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.083347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3831ac9446a01c59b9333f72ccada707-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal\" (UID: \"3831ac9446a01c59b9333f72ccada707\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:41.083397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.083366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c72a50c000c2c43d429e4d0ccd75117-config\") pod \"kube-apiserver-proxy-ip-10-0-135-96.ec2.internal\" (UID: \"6c72a50c000c2c43d429e4d0ccd75117\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:41.083506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.083412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c72a50c000c2c43d429e4d0ccd75117-config\") pod \"kube-apiserver-proxy-ip-10-0-135-96.ec2.internal\" (UID: \"6c72a50c000c2c43d429e4d0ccd75117\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:41.163617 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.163528    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:41.262077 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.262056    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:41.264155 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.264126    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:41.265781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.265764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:41.364533 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.364494    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:41.465099 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.465025    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:41.565526 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.565497    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:41.609063 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.609031    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 23:01:41.609593 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.609202    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:01:41.609593 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.609216    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:01:41.666570 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.666530    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:41.675467 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.675402    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 22:56:40 +0000 UTC" deadline="2028-05-24 23:06:34.03051591 +0000 UTC"
Sep 10 23:01:41.675467 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.675463    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14928h4m52.355056617s"
Sep 10 23:01:41.766816 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.766793    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:41.772106 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:41.772081    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c72a50c000c2c43d429e4d0ccd75117.slice/crio-24948fa8ce4b5453db7e115d41ec1b93021f7edc9db39732c2145e17d64f9ab7 WatchSource:0}: Error finding container 24948fa8ce4b5453db7e115d41ec1b93021f7edc9db39732c2145e17d64f9ab7: Status 404 returned error can't find the container with id 24948fa8ce4b5453db7e115d41ec1b93021f7edc9db39732c2145e17d64f9ab7
Sep 10 23:01:41.775419 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.775397    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:01:41.831379 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:41.831334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal" event={"ID":"6c72a50c000c2c43d429e4d0ccd75117","Type":"ContainerStarted","Data":"24948fa8ce4b5453db7e115d41ec1b93021f7edc9db39732c2145e17d64f9ab7"}
Sep 10 23:01:41.867524 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.867484    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:41.968036 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:41.968006    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:42.036568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.036487    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:42.069245 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.069204    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-96.ec2.internal\" not found"
Sep 10 23:01:42.104056 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.104030    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:42.180417 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.180373    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:42.193981 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.193956    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:01:42.195036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.195006    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal"
Sep 10 23:01:42.204977 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.204954    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:01:42.478569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.478481    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:42.495322 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:42.495274    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3831ac9446a01c59b9333f72ccada707.slice/crio-95d82e94bc39cca6c47ba6a707e038216fccd4e67072638e9d14f580041c7524 WatchSource:0}: Error finding container 95d82e94bc39cca6c47ba6a707e038216fccd4e67072638e9d14f580041c7524: Status 404 returned error can't find the container with id 95d82e94bc39cca6c47ba6a707e038216fccd4e67072638e9d14f580041c7524
Sep 10 23:01:42.658822 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.658793    2580 apiserver.go:52] "Watching apiserver"
Sep 10 23:01:42.668362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.668335    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 23:01:42.669694 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.669667    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal","openshift-multus/multus-vqtcn","openshift-network-diagnostics/network-check-target-2dj82","openshift-network-operator/iptables-alerter-9628v","kube-system/global-pull-secret-syncer-zsct5","openshift-dns/node-resolver-dh6gp","openshift-image-registry/node-ca-hs27n","openshift-multus/multus-additional-cni-plugins-crvrl","openshift-multus/network-metrics-daemon-v26f5","openshift-ovn-kubernetes/ovnkube-node-4jwww","kube-system/konnectivity-agent-2hghp","kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls","openshift-cluster-node-tuning-operator/tuned-2qmhc"]
Sep 10 23:01:42.671964 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.671942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.673798 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.673769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:01:42.674618 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.674575    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 23:01:42.674724 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.674619    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 23:01:42.674724 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.674682    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 23:01:42.674892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.674618    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 23:01:42.674892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.674840    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 23:01:42.674892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.674851    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 23:01:42.675052 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.674854    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4g5xv\""
Sep 10 23:01:42.676115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.675912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 23:01:42.676115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.675922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.676115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.675937    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v9pzx\""
Sep 10 23:01:42.676115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.675955    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 23:01:42.678106 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.678084    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 23:01:42.678235 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.678170    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:01:42.678522 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.678504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlw2p\""
Sep 10 23:01:42.680008 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.679982    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 23:01:42.680128 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.680110    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.682374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.682354    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 23:01:42.682462 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.682393    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.682883 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.682853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 23:01:42.683036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.683014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 23:01:42.683219 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.683197    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d9zbd\""
Sep 10 23:01:42.684705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.684398    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 23:01:42.684705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.684594    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 23:01:42.684888 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.684871    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 23:01:42.685100 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.685071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f4g4m\""
Sep 10 23:01:42.686729 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.686700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.688715 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.688696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:42.688824 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.688781    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:01:42.688891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.688848    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 23:01:42.688891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.688856    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 23:01:42.688891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.688885    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4ms9j\""
Sep 10 23:01:42.690967 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.690950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:42.691045 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.691019    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:01:42.692239 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a15dc057-cca1-4b54-a310-2c9d1c34c192-env-overrides\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.692344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-sysconfig\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.692344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-kubernetes\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.692344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692294    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:01:42.692344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.692344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a15dc057-cca1-4b54-a310-2c9d1c34c192-ovnkube-script-lib\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-tuned\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-tmp\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-run-systemd\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ded8dd6-e929-46c4-a995-649649748c61-agent-certs\") pod \"konnectivity-agent-2hghp\" (UID: \"4ded8dd6-e929-46c4-a995-649649748c61\") " pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ded8dd6-e929-46c4-a995-649649748c61-konnectivity-ca\") pod \"konnectivity-agent-2hghp\" (UID: \"4ded8dd6-e929-46c4-a995-649649748c61\") " pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-run\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-sys\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-host\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmw4v\" (UniqueName: \"kubernetes.io/projected/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-kube-api-access-dmw4v\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.692659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptcdn\" (UniqueName: \"kubernetes.io/projected/575fae60-bced-4137-b2c8-9555e0e59a1b-kube-api-access-ptcdn\") pod \"node-ca-hs27n\" (UID: \"575fae60-bced-4137-b2c8-9555e0e59a1b\") " pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-systemd\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/575fae60-bced-4137-b2c8-9555e0e59a1b-host\") pod \"node-ca-hs27n\" (UID: \"575fae60-bced-4137-b2c8-9555e0e59a1b\") " pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/575fae60-bced-4137-b2c8-9555e0e59a1b-serviceca\") pod \"node-ca-hs27n\" (UID: \"575fae60-bced-4137-b2c8-9555e0e59a1b\") " pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a15dc057-cca1-4b54-a310-2c9d1c34c192-ovnkube-config\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjq6w\" (UniqueName: \"kubernetes.io/projected/a15dc057-cca1-4b54-a310-2c9d1c34c192-kube-api-access-cjq6w\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-sysctl-conf\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-var-lib-kubelet\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-kubelet\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-systemd-units\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-run-openvswitch\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-run-ovn\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-lib-modules\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.692987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtrmt\" (UniqueName: \"kubernetes.io/projected/c53010f5-e404-4c38-a4b3-e9d812b81436-kube-api-access-dtrmt\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-socket-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.693038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-registration-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-sys-fs\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-slash\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-run-ovn-kubernetes\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-sysctl-d\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-device-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-run-netns\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-var-lib-openvswitch\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-node-log\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-cni-netd\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a15dc057-cca1-4b54-a310-2c9d1c34c192-ovn-node-metrics-cert\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-modprobe-d\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-etc-selinux\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-etc-openvswitch\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-log-socket\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.693832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.693492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-cni-bin\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.695139 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.695122    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 23:01:42.695266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.695223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.695851 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.695828    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 23:01:42.695851 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.695845    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 23:01:42.695973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.695884    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 23:01:42.695973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.695851    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k9sdg\""
Sep 10 23:01:42.698049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.697739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.698049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.697811    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:01:42.698049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.697820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 23:01:42.698049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.698015    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 23:01:42.698306 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.698200    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-78gpp\""
Sep 10 23:01:42.699974 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.699956    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5ktjc\""
Sep 10 23:01:42.700159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.700138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:42.700278 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.699959    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 23:01:42.700278 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.700264    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:01:42.700427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.700182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 23:01:42.716812 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.716795    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qj6kg"
Sep 10 23:01:42.725180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.725108    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qj6kg"
Sep 10 23:01:42.781554 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.781509    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 23:01:42.794706 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-var-lib-kubelet\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.794856 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-device-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.794856 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-node-log\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.794856 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-modprobe-d\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.794856 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-host\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.794856 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-node-log\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmw4v\" (UniqueName: \"kubernetes.io/projected/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-kube-api-access-dmw4v\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-host\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-etc-selinux\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-log-socket\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-cni-bin\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-device-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-tmp\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.794984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-modprobe-d\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-log-socket\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-cni-bin\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-etc-selinux\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a15dc057-cca1-4b54-a310-2c9d1c34c192-ovnkube-script-lib\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.795108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-tuned\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5652\" (UniqueName: \"kubernetes.io/projected/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-kube-api-access-c5652\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-run-systemd\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-kubelet-config\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-run-systemd\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptcdn\" (UniqueName: \"kubernetes.io/projected/575fae60-bced-4137-b2c8-9555e0e59a1b-kube-api-access-ptcdn\") pod \"node-ca-hs27n\" (UID: \"575fae60-bced-4137-b2c8-9555e0e59a1b\") " pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-systemd\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-conf-dir\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5f89\" (UniqueName: \"kubernetes.io/projected/2792cca2-0f94-438b-bf98-81a0755879a9-kube-api-access-x5f89\") pod \"iptables-alerter-9628v\" (UID: \"2792cca2-0f94-438b-bf98-81a0755879a9\") " pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-cni-binary-copy\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-systemd\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/575fae60-bced-4137-b2c8-9555e0e59a1b-host\") pod \"node-ca-hs27n\" (UID: \"575fae60-bced-4137-b2c8-9555e0e59a1b\") " pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/575fae60-bced-4137-b2c8-9555e0e59a1b-serviceca\") pod \"node-ca-hs27n\" (UID: \"575fae60-bced-4137-b2c8-9555e0e59a1b\") " pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795524    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 23:01:42.795766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/575fae60-bced-4137-b2c8-9555e0e59a1b-host\") pod \"node-ca-hs27n\" (UID: \"575fae60-bced-4137-b2c8-9555e0e59a1b\") " pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a15dc057-cca1-4b54-a310-2c9d1c34c192-ovnkube-config\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-sysctl-conf\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-var-lib-kubelet\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-kubelet\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-systemd-units\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-run-ovn\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-kubelet\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-lib-modules\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-sysctl-conf\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-systemd-units\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a15dc057-cca1-4b54-a310-2c9d1c34c192-ovnkube-script-lib\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtrmt\" (UniqueName: \"kubernetes.io/projected/c53010f5-e404-4c38-a4b3-e9d812b81436-kube-api-access-dtrmt\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-run-ovn\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-sysctl-d\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-system-cni-dir\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.796549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-cni-dir\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-socket-dir-parent\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-lib-modules\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/575fae60-bced-4137-b2c8-9555e0e59a1b-serviceca\") pod \"node-ca-hs27n\" (UID: \"575fae60-bced-4137-b2c8-9555e0e59a1b\") " pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.795994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-run-netns\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-var-lib-openvswitch\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-sysctl-d\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-run-netns\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a15dc057-cca1-4b54-a310-2c9d1c34c192-ovnkube-config\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-cni-netd\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-cni-netd\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-var-lib-openvswitch\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-var-lib-kubelet\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a15dc057-cca1-4b54-a310-2c9d1c34c192-ovn-node-metrics-cert\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-run\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.797517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-sys\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19129d1e-38ab-4ac7-b5ee-06d696663198-tmp-dir\") pod \"node-resolver-dh6gp\" (UID: \"19129d1e-38ab-4ac7-b5ee-06d696663198\") " pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-etc-openvswitch\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-run\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a15dc057-cca1-4b54-a310-2c9d1c34c192-env-overrides\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-sys\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-sysconfig\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-etc-openvswitch\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-kubernetes\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-run-k8s-cni-cncf-io\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-sysconfig\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-daemon-config\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-kubernetes\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-cnibin\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.798439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-var-lib-cni-bin\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-hostroot\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2792cca2-0f94-438b-bf98-81a0755879a9-iptables-alerter-script\") pod \"iptables-alerter-9628v\" (UID: \"2792cca2-0f94-438b-bf98-81a0755879a9\") " pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a15dc057-cca1-4b54-a310-2c9d1c34c192-env-overrides\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-system-cni-dir\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-cnibin\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ded8dd6-e929-46c4-a995-649649748c61-agent-certs\") pod \"konnectivity-agent-2hghp\" (UID: \"4ded8dd6-e929-46c4-a995-649649748c61\") " pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ded8dd6-e929-46c4-a995-649649748c61-konnectivity-ca\") pod \"konnectivity-agent-2hghp\" (UID: \"4ded8dd6-e929-46c4-a995-649649748c61\") " pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56ea190a-f9a3-48ff-89b4-6899da36d6da-cni-binary-copy\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246cn\" (UniqueName: \"kubernetes.io/projected/19129d1e-38ab-4ac7-b5ee-06d696663198-kube-api-access-246cn\") pod \"node-resolver-dh6gp\" (UID: \"19129d1e-38ab-4ac7-b5ee-06d696663198\") " pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.796968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d5fq\" (UniqueName: \"kubernetes.io/projected/6b7b8f65-5f39-4ce7-b892-c70e379f3437-kube-api-access-4d5fq\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-run-multus-certs\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjq6w\" (UniqueName: \"kubernetes.io/projected/a15dc057-cca1-4b54-a310-2c9d1c34c192-kube-api-access-cjq6w\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19129d1e-38ab-4ac7-b5ee-06d696663198-hosts-file\") pod \"node-resolver-dh6gp\" (UID: \"19129d1e-38ab-4ac7-b5ee-06d696663198\") " pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-run-netns\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-run-openvswitch\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.799362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-dbus\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-etc-kubernetes\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktwnz\" (UniqueName: \"kubernetes.io/projected/56ea190a-f9a3-48ff-89b4-6899da36d6da-kube-api-access-ktwnz\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2792cca2-0f94-438b-bf98-81a0755879a9-host-slash\") pod \"iptables-alerter-9628v\" (UID: \"2792cca2-0f94-438b-bf98-81a0755879a9\") " pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-socket-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-registration-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-sys-fs\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-slash\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-run-ovn-kubernetes\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-var-lib-cni-multus\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-run-openvswitch\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-registration-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ded8dd6-e929-46c4-a995-649649748c61-konnectivity-ca\") pod \"konnectivity-agent-2hghp\" (UID: \"4ded8dd6-e929-46c4-a995-649649748c61\") " pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-slash\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-socket-dir\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c53010f5-e404-4c38-a4b3-e9d812b81436-sys-fs\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.797986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a15dc057-cca1-4b54-a310-2c9d1c34c192-host-run-ovn-kubernetes\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.800150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.799699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-etc-tuned\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.800800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.799755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-tmp\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.800800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.799859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a15dc057-cca1-4b54-a310-2c9d1c34c192-ovn-node-metrics-cert\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.802736 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.802715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ded8dd6-e929-46c4-a995-649649748c61-agent-certs\") pod \"konnectivity-agent-2hghp\" (UID: \"4ded8dd6-e929-46c4-a995-649649748c61\") " pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:01:42.807776 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.807752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptcdn\" (UniqueName: \"kubernetes.io/projected/575fae60-bced-4137-b2c8-9555e0e59a1b-kube-api-access-ptcdn\") pod \"node-ca-hs27n\" (UID: \"575fae60-bced-4137-b2c8-9555e0e59a1b\") " pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:42.808363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.808343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmw4v\" (UniqueName: \"kubernetes.io/projected/75b65a8b-8292-43cc-9604-e2a1c21ed3a7-kube-api-access-dmw4v\") pod \"tuned-2qmhc\" (UID: \"75b65a8b-8292-43cc-9604-e2a1c21ed3a7\") " pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:42.808511 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.808493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtrmt\" (UniqueName: \"kubernetes.io/projected/c53010f5-e404-4c38-a4b3-e9d812b81436-kube-api-access-dtrmt\") pod \"aws-ebs-csi-driver-node-h5qls\" (UID: \"c53010f5-e404-4c38-a4b3-e9d812b81436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:42.809236 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.809213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjq6w\" (UniqueName: \"kubernetes.io/projected/a15dc057-cca1-4b54-a310-2c9d1c34c192-kube-api-access-cjq6w\") pod \"ovnkube-node-4jwww\" (UID: \"a15dc057-cca1-4b54-a310-2c9d1c34c192\") " pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.833707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.833678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal" event={"ID":"3831ac9446a01c59b9333f72ccada707","Type":"ContainerStarted","Data":"95d82e94bc39cca6c47ba6a707e038216fccd4e67072638e9d14f580041c7524"}
Sep 10 23:01:42.896703 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.896667    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:42.897769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-var-lib-cni-multus\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.897907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-var-lib-kubelet\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.897907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.897907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5652\" (UniqueName: \"kubernetes.io/projected/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-kube-api-access-c5652\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.897907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-kubelet-config\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:42.897907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:42.897907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-var-lib-cni-multus\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.897907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-var-lib-kubelet\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.897907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-conf-dir\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-conf-dir\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-kubelet-config\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.897977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5f89\" (UniqueName: \"kubernetes.io/projected/2792cca2-0f94-438b-bf98-81a0755879a9-kube-api-access-x5f89\") pod \"iptables-alerter-9628v\" (UID: \"2792cca2-0f94-438b-bf98-81a0755879a9\") " pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.897994    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-cni-binary-copy\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.898083    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs podName:6b7b8f65-5f39-4ce7-b892-c70e379f3437 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:43.398054444 +0000 UTC m=+3.099687371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs") pod "network-metrics-daemon-v26f5" (UID: "6b7b8f65-5f39-4ce7-b892-c70e379f3437") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-system-cni-dir\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-cni-dir\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-socket-dir-parent\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-system-cni-dir\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19129d1e-38ab-4ac7-b5ee-06d696663198-tmp-dir\") pod \"node-resolver-dh6gp\" (UID: \"19129d1e-38ab-4ac7-b5ee-06d696663198\") " pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:42.898335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-socket-dir-parent\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-run-k8s-cni-cncf-io\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-daemon-config\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-cni-dir\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-cnibin\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-var-lib-cni-bin\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-cnibin\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-hostroot\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2792cca2-0f94-438b-bf98-81a0755879a9-iptables-alerter-script\") pod \"iptables-alerter-9628v\" (UID: \"2792cca2-0f94-438b-bf98-81a0755879a9\") " pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-system-cni-dir\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-cnibin\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-cni-binary-copy\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-run-k8s-cni-cncf-io\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56ea190a-f9a3-48ff-89b4-6899da36d6da-cni-binary-copy\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.898666    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246cn\" (UniqueName: \"kubernetes.io/projected/19129d1e-38ab-4ac7-b5ee-06d696663198-kube-api-access-246cn\") pod \"node-resolver-dh6gp\" (UID: \"19129d1e-38ab-4ac7-b5ee-06d696663198\") " pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.898996 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.898830    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret podName:1071ba46-0db7-4471-8c45-33cd7ca6dc8e nodeName:}" failed. No retries permitted until 2026-09-10 23:01:43.398796851 +0000 UTC m=+3.100429792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret") pod "global-pull-secret-syncer-zsct5" (UID: "1071ba46-0db7-4471-8c45-33cd7ca6dc8e") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-cnibin\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-system-cni-dir\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-var-lib-cni-bin\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d5fq\" (UniqueName: \"kubernetes.io/projected/6b7b8f65-5f39-4ce7-b892-c70e379f3437-kube-api-access-4d5fq\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-hostroot\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.898978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-run-multus-certs\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19129d1e-38ab-4ac7-b5ee-06d696663198-hosts-file\") pod \"node-resolver-dh6gp\" (UID: \"19129d1e-38ab-4ac7-b5ee-06d696663198\") " pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-run-netns\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-dbus\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-etc-kubernetes\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwnz\" (UniqueName: \"kubernetes.io/projected/56ea190a-f9a3-48ff-89b4-6899da36d6da-kube-api-access-ktwnz\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-run-netns\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2792cca2-0f94-438b-bf98-81a0755879a9-host-slash\") pod \"iptables-alerter-9628v\" (UID: \"2792cca2-0f94-438b-bf98-81a0755879a9\") " pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56ea190a-f9a3-48ff-89b4-6899da36d6da-multus-daemon-config\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2792cca2-0f94-438b-bf98-81a0755879a9-host-slash\") pod \"iptables-alerter-9628v\" (UID: \"2792cca2-0f94-438b-bf98-81a0755879a9\") " pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-host-run-multus-certs\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.899803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56ea190a-f9a3-48ff-89b4-6899da36d6da-cni-binary-copy\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.900652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56ea190a-f9a3-48ff-89b4-6899da36d6da-etc-kubernetes\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.900652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-dbus\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:42.900652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19129d1e-38ab-4ac7-b5ee-06d696663198-hosts-file\") pod \"node-resolver-dh6gp\" (UID: \"19129d1e-38ab-4ac7-b5ee-06d696663198\") " pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.900652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2792cca2-0f94-438b-bf98-81a0755879a9-iptables-alerter-script\") pod \"iptables-alerter-9628v\" (UID: \"2792cca2-0f94-438b-bf98-81a0755879a9\") " pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.900652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.899939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19129d1e-38ab-4ac7-b5ee-06d696663198-tmp-dir\") pod \"node-resolver-dh6gp\" (UID: \"19129d1e-38ab-4ac7-b5ee-06d696663198\") " pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.907950 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.907930    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:42.908087 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.907953    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:42.908087 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.907968    2580 projected.go:196] Error preparing data for projected volume kube-api-access-87frl for pod openshift-network-diagnostics/network-check-target-2dj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:42.908087 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:42.908053    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl podName:c7efa847-6bea-4a47-a36b-13a52ae0812c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:43.408036229 +0000 UTC m=+3.109669170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-87frl" (UniqueName: "kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl") pod "network-check-target-2dj82" (UID: "c7efa847-6bea-4a47-a36b-13a52ae0812c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:42.909895 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.909839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5652\" (UniqueName: \"kubernetes.io/projected/ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69-kube-api-access-c5652\") pod \"multus-additional-cni-plugins-crvrl\" (UID: \"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69\") " pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:42.911401 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.911376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5f89\" (UniqueName: \"kubernetes.io/projected/2792cca2-0f94-438b-bf98-81a0755879a9-kube-api-access-x5f89\") pod \"iptables-alerter-9628v\" (UID: \"2792cca2-0f94-438b-bf98-81a0755879a9\") " pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:42.911934 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.911910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktwnz\" (UniqueName: \"kubernetes.io/projected/56ea190a-f9a3-48ff-89b4-6899da36d6da-kube-api-access-ktwnz\") pod \"multus-vqtcn\" (UID: \"56ea190a-f9a3-48ff-89b4-6899da36d6da\") " pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:42.912110 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.912092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-246cn\" (UniqueName: \"kubernetes.io/projected/19129d1e-38ab-4ac7-b5ee-06d696663198-kube-api-access-246cn\") pod \"node-resolver-dh6gp\" (UID: \"19129d1e-38ab-4ac7-b5ee-06d696663198\") " pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:42.912223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.912204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d5fq\" (UniqueName: \"kubernetes.io/projected/6b7b8f65-5f39-4ce7-b892-c70e379f3437-kube-api-access-4d5fq\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:42.986408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.986328    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:01:42.993027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:42.993000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:01:43.002692 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.002671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2qmhc"
Sep 10 23:01:43.006428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.006407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hs27n"
Sep 10 23:01:43.014157 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.014138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls"
Sep 10 23:01:43.020759 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.020737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dh6gp"
Sep 10 23:01:43.028489 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.028469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqtcn"
Sep 10 23:01:43.035040 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.035014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9628v"
Sep 10 23:01:43.041257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.041238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-crvrl"
Sep 10 23:01:43.402344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.402316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:43.402460 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.402361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:43.402509 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:43.402467    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:43.402509 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:43.402482    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:43.402598 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:43.402533    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs podName:6b7b8f65-5f39-4ce7-b892-c70e379f3437 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:44.402514613 +0000 UTC m=+4.104147556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs") pod "network-metrics-daemon-v26f5" (UID: "6b7b8f65-5f39-4ce7-b892-c70e379f3437") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:43.402598 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:43.402566    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret podName:1071ba46-0db7-4471-8c45-33cd7ca6dc8e nodeName:}" failed. No retries permitted until 2026-09-10 23:01:44.402557775 +0000 UTC m=+4.104190701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret") pod "global-pull-secret-syncer-zsct5" (UID: "1071ba46-0db7-4471-8c45-33cd7ca6dc8e") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:43.403810 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:43.403649    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15dc057_cca1_4b54_a310_2c9d1c34c192.slice/crio-5b7e4d303f02ccc87124b772fc67d3a84545c94d3f14d30dbdf2ba8c5f120aaa WatchSource:0}: Error finding container 5b7e4d303f02ccc87124b772fc67d3a84545c94d3f14d30dbdf2ba8c5f120aaa: Status 404 returned error can't find the container with id 5b7e4d303f02ccc87124b772fc67d3a84545c94d3f14d30dbdf2ba8c5f120aaa
Sep 10 23:01:43.405153 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:43.405135    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5a40bf_f0d6_45d7_9d2e_c0a7f1169c69.slice/crio-8e14d9b0c374f170596491ef0a692d1f44a3ef9a5ad3b52abf2c638a0e7184f8 WatchSource:0}: Error finding container 8e14d9b0c374f170596491ef0a692d1f44a3ef9a5ad3b52abf2c638a0e7184f8: Status 404 returned error can't find the container with id 8e14d9b0c374f170596491ef0a692d1f44a3ef9a5ad3b52abf2c638a0e7184f8
Sep 10 23:01:43.408497 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:43.407995    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ea190a_f9a3_48ff_89b4_6899da36d6da.slice/crio-5952de0f0fe3555adbd8c0eefb04a17c1c9544d9803bcd6c15417562691fd8e3 WatchSource:0}: Error finding container 5952de0f0fe3555adbd8c0eefb04a17c1c9544d9803bcd6c15417562691fd8e3: Status 404 returned error can't find the container with id 5952de0f0fe3555adbd8c0eefb04a17c1c9544d9803bcd6c15417562691fd8e3
Sep 10 23:01:43.410113 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:43.410095    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2792cca2_0f94_438b_bf98_81a0755879a9.slice/crio-714dc81b1657a67fbde0ec9a05a116bcbae2657bfb7d6a9dbdf6a05fc773ee9d WatchSource:0}: Error finding container 714dc81b1657a67fbde0ec9a05a116bcbae2657bfb7d6a9dbdf6a05fc773ee9d: Status 404 returned error can't find the container with id 714dc81b1657a67fbde0ec9a05a116bcbae2657bfb7d6a9dbdf6a05fc773ee9d
Sep 10 23:01:43.411431 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:43.411413    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575fae60_bced_4137_b2c8_9555e0e59a1b.slice/crio-6322c561798092342f0ae3483471b9ce1666d4cd0c9bfdf5e7fb694693eb95db WatchSource:0}: Error finding container 6322c561798092342f0ae3483471b9ce1666d4cd0c9bfdf5e7fb694693eb95db: Status 404 returned error can't find the container with id 6322c561798092342f0ae3483471b9ce1666d4cd0c9bfdf5e7fb694693eb95db
Sep 10 23:01:43.413483 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:43.413358    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ded8dd6_e929_46c4_a995_649649748c61.slice/crio-86e74b757691b154def04d9405c883b18a7b2d02cf5444f439ee851fb8d1cd58 WatchSource:0}: Error finding container 86e74b757691b154def04d9405c883b18a7b2d02cf5444f439ee851fb8d1cd58: Status 404 returned error can't find the container with id 86e74b757691b154def04d9405c883b18a7b2d02cf5444f439ee851fb8d1cd58
Sep 10 23:01:43.414108 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:43.413959    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b65a8b_8292_43cc_9604_e2a1c21ed3a7.slice/crio-6ebdd929ab4b71bce55bf38d7d924a901054239e175a7ee8192247d742bc14c2 WatchSource:0}: Error finding container 6ebdd929ab4b71bce55bf38d7d924a901054239e175a7ee8192247d742bc14c2: Status 404 returned error can't find the container with id 6ebdd929ab4b71bce55bf38d7d924a901054239e175a7ee8192247d742bc14c2
Sep 10 23:01:43.417482 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:01:43.417459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53010f5_e404_4c38_a4b3_e9d812b81436.slice/crio-d3389114455b6cec1bc4451a1eb0ca92947d77081970e5eaeb9ef014c6dd5440 WatchSource:0}: Error finding container d3389114455b6cec1bc4451a1eb0ca92947d77081970e5eaeb9ef014c6dd5440: Status 404 returned error can't find the container with id d3389114455b6cec1bc4451a1eb0ca92947d77081970e5eaeb9ef014c6dd5440
Sep 10 23:01:43.503100 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.503068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:43.503238 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:43.503221    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:43.503276 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:43.503242    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:43.503276 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:43.503251    2580 projected.go:196] Error preparing data for projected volume kube-api-access-87frl for pod openshift-network-diagnostics/network-check-target-2dj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:43.503337 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:43.503299    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl podName:c7efa847-6bea-4a47-a36b-13a52ae0812c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:44.50328536 +0000 UTC m=+4.204918284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-87frl" (UniqueName: "kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl") pod "network-check-target-2dj82" (UID: "c7efa847-6bea-4a47-a36b-13a52ae0812c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:43.728853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.728738    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:56:42 +0000 UTC" deadline="2028-05-14 23:58:02.289734161 +0000 UTC"
Sep 10 23:01:43.728853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.728800    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14688h56m18.560936521s"
Sep 10 23:01:43.840840 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.840803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal" event={"ID":"6c72a50c000c2c43d429e4d0ccd75117","Type":"ContainerStarted","Data":"4ed1afc8fe484be4f7fcfda47f9ec2891d15d1882aacef660473fbe3ff0901fa"}
Sep 10 23:01:43.844853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.844821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dh6gp" event={"ID":"19129d1e-38ab-4ac7-b5ee-06d696663198","Type":"ContainerStarted","Data":"f1e3452fd83e94bd013e528f36feccad7cd67402152c375d71d2c45a3c8279db"}
Sep 10 23:01:43.847086 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.847058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2qmhc" event={"ID":"75b65a8b-8292-43cc-9604-e2a1c21ed3a7","Type":"ContainerStarted","Data":"6ebdd929ab4b71bce55bf38d7d924a901054239e175a7ee8192247d742bc14c2"}
Sep 10 23:01:43.856464 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.855889    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-96.ec2.internal" podStartSLOduration=1.855869932 podStartE2EDuration="1.855869932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:43.854863393 +0000 UTC m=+3.556496338" watchObservedRunningTime="2026-09-10 23:01:43.855869932 +0000 UTC m=+3.557502877"
Sep 10 23:01:43.856464 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.856313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9628v" event={"ID":"2792cca2-0f94-438b-bf98-81a0755879a9","Type":"ContainerStarted","Data":"714dc81b1657a67fbde0ec9a05a116bcbae2657bfb7d6a9dbdf6a05fc773ee9d"}
Sep 10 23:01:43.861820 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.861794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqtcn" event={"ID":"56ea190a-f9a3-48ff-89b4-6899da36d6da","Type":"ContainerStarted","Data":"5952de0f0fe3555adbd8c0eefb04a17c1c9544d9803bcd6c15417562691fd8e3"}
Sep 10 23:01:43.863248 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.863224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crvrl" event={"ID":"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69","Type":"ContainerStarted","Data":"8e14d9b0c374f170596491ef0a692d1f44a3ef9a5ad3b52abf2c638a0e7184f8"}
Sep 10 23:01:43.865553 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.865514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" event={"ID":"a15dc057-cca1-4b54-a310-2c9d1c34c192","Type":"ContainerStarted","Data":"5b7e4d303f02ccc87124b772fc67d3a84545c94d3f14d30dbdf2ba8c5f120aaa"}
Sep 10 23:01:43.871612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.871578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls" event={"ID":"c53010f5-e404-4c38-a4b3-e9d812b81436","Type":"ContainerStarted","Data":"d3389114455b6cec1bc4451a1eb0ca92947d77081970e5eaeb9ef014c6dd5440"}
Sep 10 23:01:43.876693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.876646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2hghp" event={"ID":"4ded8dd6-e929-46c4-a995-649649748c61","Type":"ContainerStarted","Data":"86e74b757691b154def04d9405c883b18a7b2d02cf5444f439ee851fb8d1cd58"}
Sep 10 23:01:43.882442 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:43.882385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hs27n" event={"ID":"575fae60-bced-4137-b2c8-9555e0e59a1b","Type":"ContainerStarted","Data":"6322c561798092342f0ae3483471b9ce1666d4cd0c9bfdf5e7fb694693eb95db"}
Sep 10 23:01:44.411403 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:44.411142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:44.411403 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:44.411215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:44.411403 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.411370    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:44.411689 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.411430    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs podName:6b7b8f65-5f39-4ce7-b892-c70e379f3437 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:46.41141292 +0000 UTC m=+6.113045844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs") pod "network-metrics-daemon-v26f5" (UID: "6b7b8f65-5f39-4ce7-b892-c70e379f3437") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:44.411850 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.411829    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:44.411935 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.411885    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret podName:1071ba46-0db7-4471-8c45-33cd7ca6dc8e nodeName:}" failed. No retries permitted until 2026-09-10 23:01:46.411870437 +0000 UTC m=+6.113503363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret") pod "global-pull-secret-syncer-zsct5" (UID: "1071ba46-0db7-4471-8c45-33cd7ca6dc8e") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:44.512383 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:44.512344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:44.512601 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.512589    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:44.512674 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.512611    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:44.512674 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.512624    2580 projected.go:196] Error preparing data for projected volume kube-api-access-87frl for pod openshift-network-diagnostics/network-check-target-2dj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:44.512776 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.512683    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl podName:c7efa847-6bea-4a47-a36b-13a52ae0812c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:46.512665135 +0000 UTC m=+6.214298068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-87frl" (UniqueName: "kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl") pod "network-check-target-2dj82" (UID: "c7efa847-6bea-4a47-a36b-13a52ae0812c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:44.729479 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:44.729388    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:56:42 +0000 UTC" deadline="2028-06-12 14:35:57.038438198 +0000 UTC"
Sep 10 23:01:44.729479 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:44.729432    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15375h34m12.309011223s"
Sep 10 23:01:44.829697 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:44.829613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:44.829697 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:44.829638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:44.829908 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.829750    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:01:44.830212 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.830181    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:01:44.830323 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:44.830268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:44.830391 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:44.830357    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:01:46.431986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:46.431254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:46.431986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:46.431332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:46.431986 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.431461    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:46.431986 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.431527    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret podName:1071ba46-0db7-4471-8c45-33cd7ca6dc8e nodeName:}" failed. No retries permitted until 2026-09-10 23:01:50.431507968 +0000 UTC m=+10.133140909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret") pod "global-pull-secret-syncer-zsct5" (UID: "1071ba46-0db7-4471-8c45-33cd7ca6dc8e") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:46.431986 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.431620    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:46.431986 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.431656    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs podName:6b7b8f65-5f39-4ce7-b892-c70e379f3437 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:50.431644511 +0000 UTC m=+10.133277433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs") pod "network-metrics-daemon-v26f5" (UID: "6b7b8f65-5f39-4ce7-b892-c70e379f3437") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:46.532365 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:46.531654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:46.532365 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.531876    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:46.532365 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.531900    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:46.532365 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.531913    2580 projected.go:196] Error preparing data for projected volume kube-api-access-87frl for pod openshift-network-diagnostics/network-check-target-2dj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:46.532365 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.531972    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl podName:c7efa847-6bea-4a47-a36b-13a52ae0812c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:50.531952764 +0000 UTC m=+10.233585689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-87frl" (UniqueName: "kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl") pod "network-check-target-2dj82" (UID: "c7efa847-6bea-4a47-a36b-13a52ae0812c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:46.828340 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:46.828240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:46.828502 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.828438    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:01:46.828829 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:46.828811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:46.828927 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.828903    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:01:46.829020 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:46.829000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:46.829123 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:46.829084    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:01:48.828414 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:48.828378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:48.828414 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:48.828417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:48.828919 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:48.828492    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:01:48.828919 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:48.828382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:48.828919 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:48.828650    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:01:48.828919 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:48.828727    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:01:50.465199 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:50.464915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:50.465199 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:50.464992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:50.465199 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.465075    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:50.465199 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.465120    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:50.465199 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.465150    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret podName:1071ba46-0db7-4471-8c45-33cd7ca6dc8e nodeName:}" failed. No retries permitted until 2026-09-10 23:01:58.465127329 +0000 UTC m=+18.166760258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret") pod "global-pull-secret-syncer-zsct5" (UID: "1071ba46-0db7-4471-8c45-33cd7ca6dc8e") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:50.465199 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.465174    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs podName:6b7b8f65-5f39-4ce7-b892-c70e379f3437 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:58.465157957 +0000 UTC m=+18.166790882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs") pod "network-metrics-daemon-v26f5" (UID: "6b7b8f65-5f39-4ce7-b892-c70e379f3437") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:50.565773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:50.565664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:50.565966 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.565798    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:50.565966 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.565819    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:50.565966 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.565830    2580 projected.go:196] Error preparing data for projected volume kube-api-access-87frl for pod openshift-network-diagnostics/network-check-target-2dj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:50.565966 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.565890    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl podName:c7efa847-6bea-4a47-a36b-13a52ae0812c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:58.565870516 +0000 UTC m=+18.267503452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-87frl" (UniqueName: "kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl") pod "network-check-target-2dj82" (UID: "c7efa847-6bea-4a47-a36b-13a52ae0812c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:50.831144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:50.831022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:50.831144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:50.831066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:50.831144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:50.831066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:50.831407 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.831163    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:01:50.831407 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.831259    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:01:50.831407 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:50.831329    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:01:52.827881 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:52.827851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:52.828305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:52.827851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:52.828305 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:52.827956    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:01:52.828305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:52.827863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:52.828305 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:52.828037    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:01:52.828305 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:52.828153    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:01:54.827946 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:54.827909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:54.828398 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:54.828045    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:01:54.828398 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:54.827909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:54.828398 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:54.828132    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:01:54.828398 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:54.827912    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:54.828398 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:54.828197    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:01:56.828083 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:56.828050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:56.828083 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:56.828079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:56.828682 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:56.828054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:56.828682 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:56.828202    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:01:56.828682 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:56.828277    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:01:56.828682 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:56.828367    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:01:58.530984 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:58.530949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:58.531444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:58.531014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:58.531444 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.531123    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:58.531444 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.531126    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:58.531444 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.531187    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret podName:1071ba46-0db7-4471-8c45-33cd7ca6dc8e nodeName:}" failed. No retries permitted until 2026-09-10 23:02:14.531166228 +0000 UTC m=+34.232799166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret") pod "global-pull-secret-syncer-zsct5" (UID: "1071ba46-0db7-4471-8c45-33cd7ca6dc8e") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:58.531444 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.531205    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs podName:6b7b8f65-5f39-4ce7-b892-c70e379f3437 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:14.531198181 +0000 UTC m=+34.232831102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs") pod "network-metrics-daemon-v26f5" (UID: "6b7b8f65-5f39-4ce7-b892-c70e379f3437") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:58.631897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:58.631860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:58.632075 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.632055    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:58.632128 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.632079    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:58.632128 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.632089    2580 projected.go:196] Error preparing data for projected volume kube-api-access-87frl for pod openshift-network-diagnostics/network-check-target-2dj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:58.632195 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.632142    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl podName:c7efa847-6bea-4a47-a36b-13a52ae0812c nodeName:}" failed. No retries permitted until 2026-09-10 23:02:14.632128086 +0000 UTC m=+34.333761011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-87frl" (UniqueName: "kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl") pod "network-check-target-2dj82" (UID: "c7efa847-6bea-4a47-a36b-13a52ae0812c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:58.827662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:58.827572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:01:58.827662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:58.827633    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:01:58.827880 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:01:58.827640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:01:58.827880 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.827746    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:01:58.827993 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.827894    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:01:58.828044 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:01:58.827988    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:00.827898 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:00.827869    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:00.827898 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:00.827917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:00.828356 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:00.827981    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:02:00.828356 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:00.828011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:00.828356 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:00.828099    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:00.828356 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:00.828208    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:02:02.829106 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.828659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:02.829653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.828999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:02.829653 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:02.829218    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:02:02.829653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.829285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:02.829653 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:02.829307    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:02.829653 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:02.829361    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:02:02.931333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.931302    2580 generic.go:356] "Generic (PLEG): container finished" podID="3831ac9446a01c59b9333f72ccada707" containerID="c9606298ea5bbef31523b8ce0824dfb8f3ac3606eee826954eb92070f90ca9d1" exitCode=0
Sep 10 23:02:02.931473 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.931393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal" event={"ID":"3831ac9446a01c59b9333f72ccada707","Type":"ContainerDied","Data":"c9606298ea5bbef31523b8ce0824dfb8f3ac3606eee826954eb92070f90ca9d1"}
Sep 10 23:02:02.932942 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.932923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dh6gp" event={"ID":"19129d1e-38ab-4ac7-b5ee-06d696663198","Type":"ContainerStarted","Data":"f5a16d14b7c4ab79e78fd62a1c65724c7610b5d136bd62dab6da54767780b8e5"}
Sep 10 23:02:02.934428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.934401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2qmhc" event={"ID":"75b65a8b-8292-43cc-9604-e2a1c21ed3a7","Type":"ContainerStarted","Data":"0b36f6e752c6de42614a3597533fea65858c351ee1fff98231add72d893957b0"}
Sep 10 23:02:02.935958 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.935935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqtcn" event={"ID":"56ea190a-f9a3-48ff-89b4-6899da36d6da","Type":"ContainerStarted","Data":"dbff8231d3bc5d419dda2515633965ce328344d128a60ec93a3d0a69285cc21e"}
Sep 10 23:02:02.937366 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.937349    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69" containerID="e4396b2359b6a39321f4fa13713981161e9a2158ae3f010467c9f65b8e51536f" exitCode=0
Sep 10 23:02:02.937454 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.937417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crvrl" event={"ID":"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69","Type":"ContainerDied","Data":"e4396b2359b6a39321f4fa13713981161e9a2158ae3f010467c9f65b8e51536f"}
Sep 10 23:02:02.940037 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.939997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" event={"ID":"a15dc057-cca1-4b54-a310-2c9d1c34c192","Type":"ContainerStarted","Data":"b13839690010f73c32c520374f705ca33976a76771aecb8ae4f0ab1e77660c43"}
Sep 10 23:02:02.941940 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.941915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" event={"ID":"a15dc057-cca1-4b54-a310-2c9d1c34c192","Type":"ContainerStarted","Data":"a58fb55b2d6e4d54fb5772a78b596df508e5618d719da60fb8e534ee0f788675"}
Sep 10 23:02:02.942036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.941978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" event={"ID":"a15dc057-cca1-4b54-a310-2c9d1c34c192","Type":"ContainerStarted","Data":"254d1105c1ca1d839614b0af5ca039c95b2750cf1fbfb9b829a8cf7f4dae316b"}
Sep 10 23:02:02.943805 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.943782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls" event={"ID":"c53010f5-e404-4c38-a4b3-e9d812b81436","Type":"ContainerStarted","Data":"fcc4a261a10aff0557c2551d61583a05ad15061f569585eda1fd93203c0bf321"}
Sep 10 23:02:02.945634 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.945610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2hghp" event={"ID":"4ded8dd6-e929-46c4-a995-649649748c61","Type":"ContainerStarted","Data":"dd9f862470cee7b1089db40c65fbce81d3ffe58e258d07d6b0f632108a34b711"}
Sep 10 23:02:02.947023 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.947004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hs27n" event={"ID":"575fae60-bced-4137-b2c8-9555e0e59a1b","Type":"ContainerStarted","Data":"1c0e6e7b1df43493fd8750e14abf8e4cb00d8da57c425ec08efb6748d3de5176"}
Sep 10 23:02:02.962973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.962916    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2qmhc" podStartSLOduration=4.230947518 podStartE2EDuration="22.962900083s" totalImagesPullingTime="18.731952565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:02.962263776 +0000 UTC m=+22.663896718" watchObservedRunningTime="2026-09-10 23:02:02.962900083 +0000 UTC m=+22.664533026"
Sep 10 23:02:02.975694 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.975656    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dh6gp" podStartSLOduration=3.209642497 podStartE2EDuration="21.975639861s" totalImagesPullingTime="18.765997364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:02.975430256 +0000 UTC m=+22.677063198" watchObservedRunningTime="2026-09-10 23:02:02.975639861 +0000 UTC m=+22.677272804"
Sep 10 23:02:02.991071 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:02.991030    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vqtcn" podStartSLOduration=3.218812355 podStartE2EDuration="21.991014285s" totalImagesPullingTime="18.77220193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:02.990808246 +0000 UTC m=+22.692441189" watchObservedRunningTime="2026-09-10 23:02:02.991014285 +0000 UTC m=+22.692647228"
Sep 10 23:02:03.025707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.025665    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hs27n" podStartSLOduration=4.334854359 podStartE2EDuration="23.025649565s" totalImagesPullingTime="18.690795206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:03.02524451 +0000 UTC m=+22.726877452" watchObservedRunningTime="2026-09-10 23:02:03.025649565 +0000 UTC m=+22.727282508"
Sep 10 23:02:03.041874 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.041830    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2hghp" podStartSLOduration=4.277912673 podStartE2EDuration="23.041814375s" totalImagesPullingTime="18.763901702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:03.041471486 +0000 UTC m=+22.743104430" watchObservedRunningTime="2026-09-10 23:02:03.041814375 +0000 UTC m=+22.743447318"
Sep 10 23:02:03.144004 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.143970    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:03.219937 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.219865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:03.950868 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.950829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal" event={"ID":"3831ac9446a01c59b9333f72ccada707","Type":"ContainerStarted","Data":"87faaa15091d42d4704161f61cb6c9b302aadfc461f5fd55b89beffaac6017a9"}
Sep 10 23:02:03.952237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.952211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9628v" event={"ID":"2792cca2-0f94-438b-bf98-81a0755879a9","Type":"ContainerStarted","Data":"0e026be07077afc505b35bf4bd3295e3833ebc9ff5070517b8a9aab40182aec6"}
Sep 10 23:02:03.955206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.955172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" event={"ID":"a15dc057-cca1-4b54-a310-2c9d1c34c192","Type":"ContainerStarted","Data":"2461fa59598f5bec03c6909d165fd34fd5f4b3f009239496667a9ec545ff5854"}
Sep 10 23:02:03.955330 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.955250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" event={"ID":"a15dc057-cca1-4b54-a310-2c9d1c34c192","Type":"ContainerStarted","Data":"3d565cb4a0c95d6f7561e370c4278ddfbaa2d6fa97ad4c14dc7b46717cb3b992"}
Sep 10 23:02:03.955330 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.955302    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" event={"ID":"a15dc057-cca1-4b54-a310-2c9d1c34c192","Type":"ContainerStarted","Data":"d3f8851f0ad26c61da6132afc95bc9e8d3ed8a26734564df464fbfc737421870"}
Sep 10 23:02:03.966562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.966496    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-96.ec2.internal" podStartSLOduration=21.966482385 podStartE2EDuration="21.966482385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:03.966089528 +0000 UTC m=+23.667722473" watchObservedRunningTime="2026-09-10 23:02:03.966482385 +0000 UTC m=+23.668115327"
Sep 10 23:02:03.982574 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:03.982509    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9628v" podStartSLOduration=4.28927671 podStartE2EDuration="22.982495462s" totalImagesPullingTime="18.693218752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:03.982066186 +0000 UTC m=+23.683699130" watchObservedRunningTime="2026-09-10 23:02:03.982495462 +0000 UTC m=+23.684128404"
Sep 10 23:02:04.105226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.105162    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:04.205559 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.205471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:04.219176 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.219153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:02:04.219870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.219855    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:02:04.288842 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.288656    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 23:02:04.749090 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.748991    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T23:02:04.288836574Z","UUID":"21a41513-ddb7-4ef6-82b0-c59a4bc1a0e1","Handler":null,"Name":"","Endpoint":""}
Sep 10 23:02:04.750464 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.750445    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 23:02:04.750464 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.750472    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 23:02:04.827550 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.827515    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:04.827550 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.827531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:04.827771 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:04.827657    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:04.827771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.827678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:04.827856 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:04.827772    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:02:04.827856 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:04.827839    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:02:04.961714 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.961626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls" event={"ID":"c53010f5-e404-4c38-a4b3-e9d812b81436","Type":"ContainerStarted","Data":"e4df57f4b286f25b7f6ca2994dec4b01b0f3367ef02be86dd432f7cb9477953f"}
Sep 10 23:02:04.962155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.961858    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:02:04.962155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:04.961961    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2hghp"
Sep 10 23:02:05.017595 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:05.017469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:05.967644 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:05.967601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" event={"ID":"a15dc057-cca1-4b54-a310-2c9d1c34c192","Type":"ContainerStarted","Data":"8cd00a48bb0ac6d948692c1649e0152096f0b53cb9f543c6b1c3e5cd08540795"}
Sep 10 23:02:05.970013 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:05.969979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls" event={"ID":"c53010f5-e404-4c38-a4b3-e9d812b81436","Type":"ContainerStarted","Data":"c6fb5f42da2f93727e0e3e4b2bea0dd9e35d66506beb0abc17551edd38297f75"}
Sep 10 23:02:05.988476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:05.988380    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5qls" podStartSLOduration=4.451119345 podStartE2EDuration="25.988358338s" totalImagesPullingTime="21.537238993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:05.987514409 +0000 UTC m=+25.689147342" watchObservedRunningTime="2026-09-10 23:02:05.988358338 +0000 UTC m=+25.689991283"
Sep 10 23:02:06.053294 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:06.053264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:06.152415 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:06.152384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:06.827384 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:06.827351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:06.827588 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:06.827456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:06.827588 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:06.827483    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:02:06.827588 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:06.827533    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:02:06.827898 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:06.827880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:06.827983 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:06.827962    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:08.827632 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:08.827395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:08.827973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:08.827438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:08.827973 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:08.827669    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:02:08.827973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:08.827450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:08.827973 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:08.827729    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:08.827973 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:08.827829    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:02:08.978665 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:08.978625    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69" containerID="c9e9a97a85fd859e1d8c47100498e17d3fb762d5d94af96dadca312cc607cff9" exitCode=0
Sep 10 23:02:08.978838 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:08.978679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crvrl" event={"ID":"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69","Type":"ContainerDied","Data":"c9e9a97a85fd859e1d8c47100498e17d3fb762d5d94af96dadca312cc607cff9"}
Sep 10 23:02:08.982809 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:08.982787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" event={"ID":"a15dc057-cca1-4b54-a310-2c9d1c34c192","Type":"ContainerStarted","Data":"f4604522af7ce5fe54068b346fc8250414ffe6509b15b4ae9d9388e58f5d1c1c"}
Sep 10 23:02:08.983075 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:08.983054    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:02:08.983144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:08.983127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:02:08.983196 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:08.983163    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:02:09.002074 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:09.001772    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:02:09.002226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:09.002184    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:02:09.023415 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:09.023364    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww" podStartSLOduration=9.952223198 podStartE2EDuration="29.023348895s" totalImagesPullingTime="19.071125697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:09.022637284 +0000 UTC m=+28.724270221" watchObservedRunningTime="2026-09-10 23:02:09.023348895 +0000 UTC m=+28.724981837"
Sep 10 23:02:09.519739 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:09.519179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:09.573485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:09.573452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:09.826241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:09.826211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:09.986913 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:09.986883    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69" containerID="e3fa2d3d3a1051e23e4101b786af270263939d885cd77ef8a2983b50d6a7ad4f" exitCode=0
Sep 10 23:02:09.987641 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:09.986966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crvrl" event={"ID":"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69","Type":"ContainerDied","Data":"e3fa2d3d3a1051e23e4101b786af270263939d885cd77ef8a2983b50d6a7ad4f"}
Sep 10 23:02:10.151206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.151173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2dj82"]
Sep 10 23:02:10.151381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.151365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:10.151500 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:10.151476    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:10.154557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.154523    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v26f5"]
Sep 10 23:02:10.154693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.154673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:10.154784 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:10.154762    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:02:10.163441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.163416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zsct5"]
Sep 10 23:02:10.163587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.163518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:10.163638 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:10.163623    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:02:10.164920 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.164892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:10.263530 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.263491    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:10.663459 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.663284    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dh6gp_19129d1e-38ab-4ac7-b5ee-06d696663198/dns-node-resolver/0.log"
Sep 10 23:02:10.991313 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.991281    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69" containerID="d262efd12938186512cfcd5f152096c1ad0ca7d253a81740ee1d9be900adba4e" exitCode=0
Sep 10 23:02:10.991709 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:10.991352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crvrl" event={"ID":"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69","Type":"ContainerDied","Data":"d262efd12938186512cfcd5f152096c1ad0ca7d253a81740ee1d9be900adba4e"}
Sep 10 23:02:11.464047 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:11.464021    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hs27n_575fae60-bced-4137-b2c8-9555e0e59a1b/node-ca/0.log"
Sep 10 23:02:11.827363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:11.827319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:11.827523 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:11.827318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:11.827523 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:11.827435    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:02:11.827523 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:11.827322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:11.827523 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:11.827513    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:02:11.827715 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:11.827605    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:13.827855 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:13.827816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:13.827855 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:13.827853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:13.828332 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:13.827874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:13.828332 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:13.827955    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:02:13.828332 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:13.828094    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:13.828332 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:13.828191    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:02:14.548002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:14.547952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:14.548176 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:14.548052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:14.548176 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:14.548131    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:02:14.548176 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:14.548146    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:02:14.548339 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:14.548211    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret podName:1071ba46-0db7-4471-8c45-33cd7ca6dc8e nodeName:}" failed. No retries permitted until 2026-09-10 23:02:46.548193469 +0000 UTC m=+66.249826395 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret") pod "global-pull-secret-syncer-zsct5" (UID: "1071ba46-0db7-4471-8c45-33cd7ca6dc8e") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:02:14.548339 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:14.548230    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs podName:6b7b8f65-5f39-4ce7-b892-c70e379f3437 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:46.5482214 +0000 UTC m=+66.249854323 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs") pod "network-metrics-daemon-v26f5" (UID: "6b7b8f65-5f39-4ce7-b892-c70e379f3437") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:02:14.553064 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:14.553038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:14.649037 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:14.648940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:14.649199 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:14.649137    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:02:14.649199 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:14.649164    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:02:14.649199 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:14.649176    2580 projected.go:196] Error preparing data for projected volume kube-api-access-87frl for pod openshift-network-diagnostics/network-check-target-2dj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:02:14.649329 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:14.649249    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl podName:c7efa847-6bea-4a47-a36b-13a52ae0812c nodeName:}" failed. No retries permitted until 2026-09-10 23:02:46.649230146 +0000 UTC m=+66.350863085 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-87frl" (UniqueName: "kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl") pod "network-check-target-2dj82" (UID: "c7efa847-6bea-4a47-a36b-13a52ae0812c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:02:14.652726 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:14.652692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:15.205327 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:15.205300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:15.305809 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:15.305764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:15.828221 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:15.828188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:15.828365 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:15.828229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:15.828365 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:15.828231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:15.828365 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:15.828314    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v26f5" podUID="6b7b8f65-5f39-4ce7-b892-c70e379f3437"
Sep 10 23:02:15.828465 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:15.828364    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dj82" podUID="c7efa847-6bea-4a47-a36b-13a52ae0812c"
Sep 10 23:02:15.828498 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:15.828470    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zsct5" podUID="1071ba46-0db7-4471-8c45-33cd7ca6dc8e"
Sep 10 23:02:16.006803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.006766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crvrl" event={"ID":"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69","Type":"ContainerStarted","Data":"61d0d23fb6e2354c588e750f146cc447d409009f34826d70351dea1e0dea461a"}
Sep 10 23:02:16.038022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.037994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:16.086607 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.086578    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-96.ec2.internal" event="NodeReady"
Sep 10 23:02:16.086771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.086740    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 23:02:16.132512 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.130069    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:16.132512 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.130399    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hstjw"]
Sep 10 23:02:16.140767 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.140740    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vmqvh"]
Sep 10 23:02:16.140907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.140877    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.142878 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.142856    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 23:02:16.143002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.142861    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-497ss\""
Sep 10 23:02:16.143122 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.143110    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 23:02:16.154813 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.154788    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hstjw"]
Sep 10 23:02:16.154923 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.154825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vmqvh"]
Sep 10 23:02:16.154923 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.154842    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wvvkj"]
Sep 10 23:02:16.154923 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.154892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vmqvh"
Sep 10 23:02:16.157257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.157201    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 23:02:16.157427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.157402    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 23:02:16.157577 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.157453    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tqzlq\""
Sep 10 23:02:16.157577 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.157465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 23:02:16.176887 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.176860    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvvkj"]
Sep 10 23:02:16.177033 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.176974    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.179027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.179009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w4w8p\""
Sep 10 23:02:16.179142 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.179045    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 23:02:16.179270 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.179252    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 23:02:16.179534 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.179519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 23:02:16.179682 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.179666    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 23:02:16.263616 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvcnc\" (UniqueName: \"kubernetes.io/projected/106e6c90-7df4-4ff0-954d-81ec5c2b397e-kube-api-access-cvcnc\") pod \"ingress-canary-vmqvh\" (UID: \"106e6c90-7df4-4ff0-954d-81ec5c2b397e\") " pod="openshift-ingress-canary/ingress-canary-vmqvh"
Sep 10 23:02:16.263616 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkz92\" (UniqueName: \"kubernetes.io/projected/399acb8d-754f-4775-a5f8-508412508334-kube-api-access-qkz92\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.264017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.264017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.264017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6204e483-a2f1-4263-ae88-066b9e5d34ea-tmp-dir\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.264017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6204e483-a2f1-4263-ae88-066b9e5d34ea-config-volume\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.264017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/399acb8d-754f-4775-a5f8-508412508334-crio-socket\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.264017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/399acb8d-754f-4775-a5f8-508412508334-data-volume\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.264017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399acb8d-754f-4775-a5f8-508412508334-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.264017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhfm\" (UniqueName: \"kubernetes.io/projected/6204e483-a2f1-4263-ae88-066b9e5d34ea-kube-api-access-4dhfm\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.264017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.263940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/106e6c90-7df4-4ff0-954d-81ec5c2b397e-cert\") pod \"ingress-canary-vmqvh\" (UID: \"106e6c90-7df4-4ff0-954d-81ec5c2b397e\") " pod="openshift-ingress-canary/ingress-canary-vmqvh"
Sep 10 23:02:16.318017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.317975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:16.365048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.365048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.365265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6204e483-a2f1-4263-ae88-066b9e5d34ea-tmp-dir\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.365265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6204e483-a2f1-4263-ae88-066b9e5d34ea-config-volume\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.365265 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:16.365167    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:16.365265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/399acb8d-754f-4775-a5f8-508412508334-crio-socket\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.365265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/399acb8d-754f-4775-a5f8-508412508334-data-volume\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.365265 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:16.365232    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls podName:6204e483-a2f1-4263-ae88-066b9e5d34ea nodeName:}" failed. No retries permitted until 2026-09-10 23:02:16.865212237 +0000 UTC m=+36.566845176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls") pod "dns-default-hstjw" (UID: "6204e483-a2f1-4263-ae88-066b9e5d34ea") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:16.365265 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:16.365253    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:16.365658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399acb8d-754f-4775-a5f8-508412508334-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.365658 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:16.365368    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls podName:399acb8d-754f-4775-a5f8-508412508334 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:16.865349687 +0000 UTC m=+36.566982611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvkj" (UID: "399acb8d-754f-4775-a5f8-508412508334") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:16.365658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/399acb8d-754f-4775-a5f8-508412508334-crio-socket\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.365658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhfm\" (UniqueName: \"kubernetes.io/projected/6204e483-a2f1-4263-ae88-066b9e5d34ea-kube-api-access-4dhfm\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.365658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/106e6c90-7df4-4ff0-954d-81ec5c2b397e-cert\") pod \"ingress-canary-vmqvh\" (UID: \"106e6c90-7df4-4ff0-954d-81ec5c2b397e\") " pod="openshift-ingress-canary/ingress-canary-vmqvh"
Sep 10 23:02:16.365658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvcnc\" (UniqueName: \"kubernetes.io/projected/106e6c90-7df4-4ff0-954d-81ec5c2b397e-kube-api-access-cvcnc\") pod \"ingress-canary-vmqvh\" (UID: \"106e6c90-7df4-4ff0-954d-81ec5c2b397e\") " pod="openshift-ingress-canary/ingress-canary-vmqvh"
Sep 10 23:02:16.365658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkz92\" (UniqueName: \"kubernetes.io/projected/399acb8d-754f-4775-a5f8-508412508334-kube-api-access-qkz92\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.365658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/399acb8d-754f-4775-a5f8-508412508334-data-volume\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.365658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6204e483-a2f1-4263-ae88-066b9e5d34ea-tmp-dir\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.366097 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399acb8d-754f-4775-a5f8-508412508334-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.366097 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.365882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6204e483-a2f1-4263-ae88-066b9e5d34ea-config-volume\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.370131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.370108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/106e6c90-7df4-4ff0-954d-81ec5c2b397e-cert\") pod \"ingress-canary-vmqvh\" (UID: \"106e6c90-7df4-4ff0-954d-81ec5c2b397e\") " pod="openshift-ingress-canary/ingress-canary-vmqvh"
Sep 10 23:02:16.370315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.370295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:16.376002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.375971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhfm\" (UniqueName: \"kubernetes.io/projected/6204e483-a2f1-4263-ae88-066b9e5d34ea-kube-api-access-4dhfm\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.376121 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.376041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkz92\" (UniqueName: \"kubernetes.io/projected/399acb8d-754f-4775-a5f8-508412508334-kube-api-access-qkz92\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.376194 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.376173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvcnc\" (UniqueName: \"kubernetes.io/projected/106e6c90-7df4-4ff0-954d-81ec5c2b397e-kube-api-access-cvcnc\") pod \"ingress-canary-vmqvh\" (UID: \"106e6c90-7df4-4ff0-954d-81ec5c2b397e\") " pod="openshift-ingress-canary/ingress-canary-vmqvh"
Sep 10 23:02:16.463162 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.463065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vmqvh"
Sep 10 23:02:16.798385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.798295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vmqvh"]
Sep 10 23:02:16.802574 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:16.802530    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106e6c90_7df4_4ff0_954d_81ec5c2b397e.slice/crio-7516c7cd1cafb6ed7b5359244ecd2248fc08f492e14d387e8b5075dbe2485c69 WatchSource:0}: Error finding container 7516c7cd1cafb6ed7b5359244ecd2248fc08f492e14d387e8b5075dbe2485c69: Status 404 returned error can't find the container with id 7516c7cd1cafb6ed7b5359244ecd2248fc08f492e14d387e8b5075dbe2485c69
Sep 10 23:02:16.869109 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.869074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:16.869109 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:16.869111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:16.869324 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:16.869228    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:16.869324 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:16.869247    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:16.869324 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:16.869296    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls podName:399acb8d-754f-4775-a5f8-508412508334 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:17.869276747 +0000 UTC m=+37.570909668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvkj" (UID: "399acb8d-754f-4775-a5f8-508412508334") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:16.869324 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:16.869312    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls podName:6204e483-a2f1-4263-ae88-066b9e5d34ea nodeName:}" failed. No retries permitted until 2026-09-10 23:02:17.86930388 +0000 UTC m=+37.570936806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls") pod "dns-default-hstjw" (UID: "6204e483-a2f1-4263-ae88-066b9e5d34ea") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:17.011899 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.011865    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69" containerID="61d0d23fb6e2354c588e750f146cc447d409009f34826d70351dea1e0dea461a" exitCode=0
Sep 10 23:02:17.012084 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.011956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crvrl" event={"ID":"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69","Type":"ContainerDied","Data":"61d0d23fb6e2354c588e750f146cc447d409009f34826d70351dea1e0dea461a"}
Sep 10 23:02:17.013178 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.013153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vmqvh" event={"ID":"106e6c90-7df4-4ff0-954d-81ec5c2b397e","Type":"ContainerStarted","Data":"7516c7cd1cafb6ed7b5359244ecd2248fc08f492e14d387e8b5075dbe2485c69"}
Sep 10 23:02:17.828222 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.828177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:17.828222 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.828211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:17.828222 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.828223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:17.831309 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.831291    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:02:17.831796 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.831778    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:02:17.831888 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.831818    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcwbm\""
Sep 10 23:02:17.832059 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.832041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:02:17.832113 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.832090    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-glltp\""
Sep 10 23:02:17.832163 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.832129    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:02:17.877851 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.877814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:17.878049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.877863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:17.878049 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:17.877990    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:17.878049 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:17.878012    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:17.878210 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:17.878073    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls podName:399acb8d-754f-4775-a5f8-508412508334 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:19.878051749 +0000 UTC m=+39.579684684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvkj" (UID: "399acb8d-754f-4775-a5f8-508412508334") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:17.878210 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:17.878098    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls podName:6204e483-a2f1-4263-ae88-066b9e5d34ea nodeName:}" failed. No retries permitted until 2026-09-10 23:02:19.878090502 +0000 UTC m=+39.579723423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls") pod "dns-default-hstjw" (UID: "6204e483-a2f1-4263-ae88-066b9e5d34ea") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:17.929638 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.929596    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-774fbc58bd-lmddv"]
Sep 10 23:02:17.948369 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.948283    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-774fbc58bd-lmddv"]
Sep 10 23:02:17.948516 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.948394    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:17.950866 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.950733    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-78nrb\""
Sep 10 23:02:17.951049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.951030    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 23:02:17.951386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.951243    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 23:02:17.951386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.951314    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 23:02:17.956339 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:17.956312    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 23:02:18.019454 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.019414    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69" containerID="d5a58f44aefa1714b1e74ddf9ae43ba8b35391c437ee79dbfafcef6c2df29216" exitCode=0
Sep 10 23:02:18.019652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.019481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crvrl" event={"ID":"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69","Type":"ContainerDied","Data":"d5a58f44aefa1714b1e74ddf9ae43ba8b35391c437ee79dbfafcef6c2df29216"}
Sep 10 23:02:18.080755 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.080671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.080916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.080772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11b68472-05eb-4f97-b418-fa51c8b4a4d3-ca-trust-extracted\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.080916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.080887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-installation-pull-secrets\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.081027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.080935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-bound-sa-token\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.081077 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.081033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-trusted-ca\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.081077 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.081062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjdv\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-kube-api-access-ndjdv\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.081180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.081091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-image-registry-private-configuration\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.081180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.081119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-certificates\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.182532 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.182490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-installation-pull-secrets\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.182740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.182590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-bound-sa-token\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.182740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.182717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-trusted-ca\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.182829 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.182752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjdv\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-kube-api-access-ndjdv\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.182829 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.182786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-image-registry-private-configuration\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.182829 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.182810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-certificates\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.182972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.182885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.182972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.182946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11b68472-05eb-4f97-b418-fa51c8b4a4d3-ca-trust-extracted\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.183071 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:18.183047    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:02:18.183071 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:18.183062    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-774fbc58bd-lmddv: secret "image-registry-tls" not found
Sep 10 23:02:18.183165 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:18.183123    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls podName:11b68472-05eb-4f97-b418-fa51c8b4a4d3 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:18.683102711 +0000 UTC m=+38.384735666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls") pod "image-registry-774fbc58bd-lmddv" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3") : secret "image-registry-tls" not found
Sep 10 23:02:18.183328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.183300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11b68472-05eb-4f97-b418-fa51c8b4a4d3-ca-trust-extracted\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.183457 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.183433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-certificates\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.183703 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.183627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-trusted-ca\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.185664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.185638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-image-registry-private-configuration\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.185767 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.185704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-installation-pull-secrets\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.193755 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.193724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-bound-sa-token\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.193916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.193810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjdv\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-kube-api-access-ndjdv\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.688621 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:18.688375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:18.688829 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:18.688534    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:02:18.688829 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:18.688694    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-774fbc58bd-lmddv: secret "image-registry-tls" not found
Sep 10 23:02:18.688829 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:18.688773    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls podName:11b68472-05eb-4f97-b418-fa51c8b4a4d3 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:19.688750358 +0000 UTC m=+39.390383278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls") pod "image-registry-774fbc58bd-lmddv" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3") : secret "image-registry-tls" not found
Sep 10 23:02:19.030107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:19.030000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crvrl" event={"ID":"ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69","Type":"ContainerStarted","Data":"0798a9ccf0c23b51a28d36d741cbceaaa6d2f16b8f1e8f73d04a3d5b03123e18"}
Sep 10 23:02:19.054209 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:19.054145    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-crvrl" podStartSLOduration=7.87231393 podStartE2EDuration="38.054122888s" totalImagesPullingTime="30.181808958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:19.052711377 +0000 UTC m=+38.754344323" watchObservedRunningTime="2026-09-10 23:02:19.054122888 +0000 UTC m=+38.755755832"
Sep 10 23:02:19.175358 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:19.175310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:19.239561 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:19.239506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:19.697797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:19.697760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:19.697955 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:19.697931    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:02:19.698001 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:19.697957    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-774fbc58bd-lmddv: secret "image-registry-tls" not found
Sep 10 23:02:19.698038 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:19.698013    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls podName:11b68472-05eb-4f97-b418-fa51c8b4a4d3 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:21.69799637 +0000 UTC m=+41.399629290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls") pod "image-registry-774fbc58bd-lmddv" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3") : secret "image-registry-tls" not found
Sep 10 23:02:19.899258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:19.899218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:19.899258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:19.899259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:19.899424 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:19.899371    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:19.899424 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:19.899378    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:19.899424 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:19.899424    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls podName:399acb8d-754f-4775-a5f8-508412508334 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:23.899410949 +0000 UTC m=+43.601043874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvkj" (UID: "399acb8d-754f-4775-a5f8-508412508334") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:19.899525 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:19.899436    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls podName:6204e483-a2f1-4263-ae88-066b9e5d34ea nodeName:}" failed. No retries permitted until 2026-09-10 23:02:23.899430887 +0000 UTC m=+43.601063807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls") pod "dns-default-hstjw" (UID: "6204e483-a2f1-4263-ae88-066b9e5d34ea") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:20.034148 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:20.034057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vmqvh" event={"ID":"106e6c90-7df4-4ff0-954d-81ec5c2b397e","Type":"ContainerStarted","Data":"18f2cc1b3319b5d6ee32b1647bf2819a30ed7d0e73f46341a5da0f817567a952"}
Sep 10 23:02:20.049223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:20.049171    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vmqvh" podStartSLOduration=1.4502836989999999 podStartE2EDuration="4.049153184s" totalImagesPullingTime="2.598869485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:20.048556321 +0000 UTC m=+39.750189257" watchObservedRunningTime="2026-09-10 23:02:20.049153184 +0000 UTC m=+39.750786124"
Sep 10 23:02:20.274171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:20.274107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:20.333502 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:20.333422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:20.598457 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:20.598419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:20.699594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:20.699556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:21.670861 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:21.670834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:21.714714 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:21.714679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:21.714835 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:21.714821    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:02:21.714885 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:21.714838    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-774fbc58bd-lmddv: secret "image-registry-tls" not found
Sep 10 23:02:21.714923 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:21.714888    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls podName:11b68472-05eb-4f97-b418-fa51c8b4a4d3 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:25.714873846 +0000 UTC m=+45.416506771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls") pod "image-registry-774fbc58bd-lmddv" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3") : secret "image-registry-tls" not found
Sep 10 23:02:21.771179 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:21.771149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:21.849852 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:21.849813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:22.047598 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.047520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:22.066600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.066563    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs"]
Sep 10 23:02:22.102170 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.102131    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs"]
Sep 10 23:02:22.102342 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.102264    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs"
Sep 10 23:02:22.104846 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.104813    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 23:02:22.104964 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.104821    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 23:02:22.105088 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.105065    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w92v6\""
Sep 10 23:02:22.120843 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.120816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:22.177876 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.177847    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:22.219321 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.219291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnft4\" (UniqueName: \"kubernetes.io/projected/54c128a2-2ed8-40e7-bd25-010f892e9f63-kube-api-access-lnft4\") pod \"migrator-8b4d697c5-4bkhs\" (UID: \"54c128a2-2ed8-40e7-bd25-010f892e9f63\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs"
Sep 10 23:02:22.280295 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.280264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:22.320435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.320353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnft4\" (UniqueName: \"kubernetes.io/projected/54c128a2-2ed8-40e7-bd25-010f892e9f63-kube-api-access-lnft4\") pod \"migrator-8b4d697c5-4bkhs\" (UID: \"54c128a2-2ed8-40e7-bd25-010f892e9f63\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs"
Sep 10 23:02:22.329832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.329799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnft4\" (UniqueName: \"kubernetes.io/projected/54c128a2-2ed8-40e7-bd25-010f892e9f63-kube-api-access-lnft4\") pod \"migrator-8b4d697c5-4bkhs\" (UID: \"54c128a2-2ed8-40e7-bd25-010f892e9f63\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs"
Sep 10 23:02:22.411036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.410996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs"
Sep 10 23:02:22.477673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.477617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:22.577885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.577782    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:22.697439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:22.697410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs"]
Sep 10 23:02:22.700719 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:22.700688    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c128a2_2ed8_40e7_bd25_010f892e9f63.slice/crio-45be3ddc5e5580a85db5f38570460dcb1a4b7cd78f47e058340d3fa33c25666e WatchSource:0}: Error finding container 45be3ddc5e5580a85db5f38570460dcb1a4b7cd78f47e058340d3fa33c25666e: Status 404 returned error can't find the container with id 45be3ddc5e5580a85db5f38570460dcb1a4b7cd78f47e058340d3fa33c25666e
Sep 10 23:02:23.041380 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:23.041350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs" event={"ID":"54c128a2-2ed8-40e7-bd25-010f892e9f63","Type":"ContainerStarted","Data":"45be3ddc5e5580a85db5f38570460dcb1a4b7cd78f47e058340d3fa33c25666e"}
Sep 10 23:02:23.934573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:23.934512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:23.934573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:23.934567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:23.934997 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:23.934662    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:23.934997 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:23.934728    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:23.934997 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:23.934738    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls podName:6204e483-a2f1-4263-ae88-066b9e5d34ea nodeName:}" failed. No retries permitted until 2026-09-10 23:02:31.934721886 +0000 UTC m=+51.636354807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls") pod "dns-default-hstjw" (UID: "6204e483-a2f1-4263-ae88-066b9e5d34ea") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:23.934997 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:23.934779    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls podName:399acb8d-754f-4775-a5f8-508412508334 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:31.934765287 +0000 UTC m=+51.636398209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvkj" (UID: "399acb8d-754f-4775-a5f8-508412508334") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:24.076728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.076703    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-b44w2"]
Sep 10 23:02:24.092715 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.092687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-b44w2"]
Sep 10 23:02:24.092859 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.092800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.094813 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.094791    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 23:02:24.095384 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.095366    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5cx2j\""
Sep 10 23:02:24.095502 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.095380    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 23:02:24.095502 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.095366    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 23:02:24.095502 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.095458    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 23:02:24.111962 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.111941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:24.163573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.163525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:24.236898 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.236818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/88ac6439-eb6a-46a4-b7c4-a3dd52148ee2-signing-cabundle\") pod \"service-ca-568869fdc5-b44w2\" (UID: \"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2\") " pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.236898 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.236863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/88ac6439-eb6a-46a4-b7c4-a3dd52148ee2-signing-key\") pod \"service-ca-568869fdc5-b44w2\" (UID: \"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2\") " pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.237077 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.236949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp9bg\" (UniqueName: \"kubernetes.io/projected/88ac6439-eb6a-46a4-b7c4-a3dd52148ee2-kube-api-access-xp9bg\") pod \"service-ca-568869fdc5-b44w2\" (UID: \"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2\") " pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.337431 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.337403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/88ac6439-eb6a-46a4-b7c4-a3dd52148ee2-signing-cabundle\") pod \"service-ca-568869fdc5-b44w2\" (UID: \"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2\") " pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.337431 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.337434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/88ac6439-eb6a-46a4-b7c4-a3dd52148ee2-signing-key\") pod \"service-ca-568869fdc5-b44w2\" (UID: \"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2\") " pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.337572 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.337481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp9bg\" (UniqueName: \"kubernetes.io/projected/88ac6439-eb6a-46a4-b7c4-a3dd52148ee2-kube-api-access-xp9bg\") pod \"service-ca-568869fdc5-b44w2\" (UID: \"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2\") " pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.338029 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.338010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/88ac6439-eb6a-46a4-b7c4-a3dd52148ee2-signing-cabundle\") pod \"service-ca-568869fdc5-b44w2\" (UID: \"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2\") " pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.339893 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.339873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/88ac6439-eb6a-46a4-b7c4-a3dd52148ee2-signing-key\") pod \"service-ca-568869fdc5-b44w2\" (UID: \"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2\") " pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.345802 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.345779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp9bg\" (UniqueName: \"kubernetes.io/projected/88ac6439-eb6a-46a4-b7c4-a3dd52148ee2-kube-api-access-xp9bg\") pod \"service-ca-568869fdc5-b44w2\" (UID: \"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2\") " pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.402476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.402440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-b44w2"
Sep 10 23:02:24.696018 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:24.695983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-b44w2"]
Sep 10 23:02:24.699667 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:24.699640    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ac6439_eb6a_46a4_b7c4_a3dd52148ee2.slice/crio-cc85387b17209eae859304b47bdfa2eb116e17de78e0bb2bbebe4eccd6463535 WatchSource:0}: Error finding container cc85387b17209eae859304b47bdfa2eb116e17de78e0bb2bbebe4eccd6463535: Status 404 returned error can't find the container with id cc85387b17209eae859304b47bdfa2eb116e17de78e0bb2bbebe4eccd6463535
Sep 10 23:02:25.047457 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:25.047424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs" event={"ID":"54c128a2-2ed8-40e7-bd25-010f892e9f63","Type":"ContainerStarted","Data":"113acdca95aaa8803c82dd3579d1d1dad86af75c8c9c6742f3cdda070d3698fd"}
Sep 10 23:02:25.047924 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:25.047472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs" event={"ID":"54c128a2-2ed8-40e7-bd25-010f892e9f63","Type":"ContainerStarted","Data":"9ccc4e03890a69bc6cc8a54aa17b6df77ceca490f5803fe71ee06fed2345b383"}
Sep 10 23:02:25.048663 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:25.048644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-b44w2" event={"ID":"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2","Type":"ContainerStarted","Data":"cc85387b17209eae859304b47bdfa2eb116e17de78e0bb2bbebe4eccd6463535"}
Sep 10 23:02:25.067146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:25.067093    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-4bkhs" podStartSLOduration=1.450793295 podStartE2EDuration="3.067077133s" totalImagesPullingTime="1.616283838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:25.066462773 +0000 UTC m=+44.768095716" watchObservedRunningTime="2026-09-10 23:02:25.067077133 +0000 UTC m=+44.768710092"
Sep 10 23:02:25.072082 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:25.072060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:25.749079 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:25.749041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:25.749258 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:25.749229    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:02:25.749258 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:25.749251    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-774fbc58bd-lmddv: secret "image-registry-tls" not found
Sep 10 23:02:25.749378 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:25.749323    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls podName:11b68472-05eb-4f97-b418-fa51c8b4a4d3 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:33.749302511 +0000 UTC m=+53.450935445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls") pod "image-registry-774fbc58bd-lmddv" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3") : secret "image-registry-tls" not found
Sep 10 23:02:27.055875 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:27.055835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-b44w2" event={"ID":"88ac6439-eb6a-46a4-b7c4-a3dd52148ee2","Type":"ContainerStarted","Data":"20ef79b4ad7ffe6f575c5d884bea53ad671439e3760f52a90f4b0b07d2242d59"}
Sep 10 23:02:27.073282 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:27.073233    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-b44w2" podStartSLOduration=0.841674177 podStartE2EDuration="3.073217988s" totalImagesPullingTime="2.231543811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:27.072013215 +0000 UTC m=+46.773646188" watchObservedRunningTime="2026-09-10 23:02:27.073217988 +0000 UTC m=+46.774850930"
Sep 10 23:02:27.642553 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:27.642518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:28.057151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:28.057058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:28.176939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:28.176901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:28.277910 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:28.277877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:28.378767 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:28.378730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:28.479417 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:28.479377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:28.953926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:28.953891    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:29.758747 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:29.758715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:29.951740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:29.951706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:30.350555 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:30.350506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:30.558711 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:30.558675    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:30.755168 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:30.755081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:31.431633 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:31.431593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:31.729832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:31.729742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:31.788478 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:31.788441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:31.957348 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:31.957294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:31.994962 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:31.994871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:31.994962 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:31.994908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:31.995186 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:31.995025    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:31.995186 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:31.995083    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls podName:399acb8d-754f-4775-a5f8-508412508334 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:47.995068686 +0000 UTC m=+67.696701611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvkj" (UID: "399acb8d-754f-4775-a5f8-508412508334") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:31.997455 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:31.997428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6204e483-a2f1-4263-ae88-066b9e5d34ea-metrics-tls\") pod \"dns-default-hstjw\" (UID: \"6204e483-a2f1-4263-ae88-066b9e5d34ea\") " pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:32.051041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:32.051004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:32.155753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:32.155674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:32.356206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:32.356174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:32.356453 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:32.356391    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hstjw"]
Sep 10 23:02:32.558000 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:32.557971    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:32.613220 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:32.613138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:32.752037 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:32.752004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:33.072564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:33.072457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hstjw" event={"ID":"6204e483-a2f1-4263-ae88-066b9e5d34ea","Type":"ContainerStarted","Data":"085410a637c6106b92d03ef07877241a8476aa808fb18a2835c222bb6631b081"}
Sep 10 23:02:33.808699 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:33.808651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:33.811518 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:33.811492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") pod \"image-registry-774fbc58bd-lmddv\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") " pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:33.854022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:33.853992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:33.863890 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:33.863857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:33.981501 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:33.981464    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:34.165970 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:34.165934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:34.353800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:34.352780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:34.469831 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:34.469800    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-774fbc58bd-lmddv"]
Sep 10 23:02:34.473165 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:34.473130    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b68472_05eb_4f97_b418_fa51c8b4a4d3.slice/crio-82968fa1e93706eb92d846b5d643fa2091cd855c5d86c097a73efbd782fa5780 WatchSource:0}: Error finding container 82968fa1e93706eb92d846b5d643fa2091cd855c5d86c097a73efbd782fa5780: Status 404 returned error can't find the container with id 82968fa1e93706eb92d846b5d643fa2091cd855c5d86c097a73efbd782fa5780
Sep 10 23:02:34.554859 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:34.554796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:34.749603 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:34.749502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:35.083980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:35.083873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hstjw" event={"ID":"6204e483-a2f1-4263-ae88-066b9e5d34ea","Type":"ContainerStarted","Data":"ab4edb536bc8808cbdc0c68b46e5916bc9c0d1b361ad6a445dbcc6d69762904c"}
Sep 10 23:02:35.083980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:35.083954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hstjw" event={"ID":"6204e483-a2f1-4263-ae88-066b9e5d34ea","Type":"ContainerStarted","Data":"74d944a18977899dd0fc166cdeaaee327d8d3ef0361acf19a0c7d67ea891d2d1"}
Sep 10 23:02:35.083980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:35.083981    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:35.085458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:35.085435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv" event={"ID":"11b68472-05eb-4f97-b418-fa51c8b4a4d3","Type":"ContainerStarted","Data":"cbef6073cff89de494be4ae14d132662bb4271751fda01db4de6dfed5d6ac0c6"}
Sep 10 23:02:35.085572 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:35.085474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv" event={"ID":"11b68472-05eb-4f97-b418-fa51c8b4a4d3","Type":"ContainerStarted","Data":"82968fa1e93706eb92d846b5d643fa2091cd855c5d86c097a73efbd782fa5780"}
Sep 10 23:02:35.085572 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:35.085513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:35.100780 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:35.100732    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hstjw" podStartSLOduration=17.229267303 podStartE2EDuration="19.100716613s" totalImagesPullingTime="1.87144931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:35.099911478 +0000 UTC m=+54.801544426" watchObservedRunningTime="2026-09-10 23:02:35.100716613 +0000 UTC m=+54.802349556"
Sep 10 23:02:35.118214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:35.118164    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv" podStartSLOduration=18.11814625 podStartE2EDuration="18.11814625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:35.117673057 +0000 UTC m=+54.819306001" watchObservedRunningTime="2026-09-10 23:02:35.11814625 +0000 UTC m=+54.819779194"
Sep 10 23:02:35.753315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:35.753279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:36.152358 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:36.152322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:36.350640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:36.350604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:36.558015 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:36.557927    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:36.755502 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:36.755467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:37.751736 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:37.751700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:37.957931 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:37.957877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:38.353873 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:38.353844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:38.559007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:38.558970    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:38.758383 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:38.758288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:40.134189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:40.134154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:40.233765 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:40.233730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:40.287057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:40.287021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:40.523960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:40.523868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:41.003977 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:41.003940    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4jwww"
Sep 10 23:02:41.157705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:41.157673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:41.258261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:41.258189    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:41.359495 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:41.359466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:41.458738 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:41.458711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:41.628513 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:41.628488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:41.730047 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:41.730019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:42.055062 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:42.054988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:42.156310 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:42.156280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:42.455457 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:42.455429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:43.036640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:43.036603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:43.178834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:43.178805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:43.278367 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:43.278328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:44.292650 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.292614    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-774fbc58bd-lmddv"]
Sep 10 23:02:44.316128 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.316097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:44.318220 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.318195    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77d46bf7c-dtlxq"]
Sep 10 23:02:44.321713 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.321698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.334004 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.333977    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d46bf7c-dtlxq"]
Sep 10 23:02:44.342684 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.342657    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"]
Sep 10 23:02:44.344619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.344598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"
Sep 10 23:02:44.348556 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.348517    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 23:02:44.348683 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.348664    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-t68b8"]
Sep 10 23:02:44.349028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.349010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mtpfj\""
Sep 10 23:02:44.349331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.349315    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 23:02:44.350984 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.350965    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"]
Sep 10 23:02:44.351100 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.351083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-t68b8"
Sep 10 23:02:44.353180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.353165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"
Sep 10 23:02:44.353309 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.353289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c5fjr\""
Sep 10 23:02:44.353741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.353723    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 23:02:44.353822 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.353773    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 23:02:44.355300 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.355283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 23:02:44.355391 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.355371    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lgxrq\""
Sep 10 23:02:44.359342 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.359318    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"]
Sep 10 23:02:44.362063 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.362042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-t68b8"]
Sep 10 23:02:44.369237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.369217    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"]
Sep 10 23:02:44.392066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f80e44d-eb36-4008-9f71-89627dc96a5e-image-registry-private-configuration\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.392066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f80e44d-eb36-4008-9f71-89627dc96a5e-installation-pull-secrets\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.392311 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f80e44d-eb36-4008-9f71-89627dc96a5e-registry-tls\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.392311 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f80e44d-eb36-4008-9f71-89627dc96a5e-bound-sa-token\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.392311 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6szxs\" (UniqueName: \"kubernetes.io/projected/8f80e44d-eb36-4008-9f71-89627dc96a5e-kube-api-access-6szxs\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.392311 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4fae47ba-f91a-48dd-8986-2649ff5f023c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ntjzf\" (UID: \"4fae47ba-f91a-48dd-8986-2649ff5f023c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"
Sep 10 23:02:44.392311 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7brdn\" (UniqueName: \"kubernetes.io/projected/6725a188-ca0f-4af0-832f-172a87277628-kube-api-access-7brdn\") pod \"downloads-76f99584f4-t68b8\" (UID: \"6725a188-ca0f-4af0-832f-172a87277628\") " pod="openshift-console/downloads-76f99584f4-t68b8"
Sep 10 23:02:44.392576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f80e44d-eb36-4008-9f71-89627dc96a5e-trusted-ca\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.392576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ad9cd3-c937-4c4a-bd49-ee4665ce8627-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p6sdw\" (UID: \"38ad9cd3-c937-4c4a-bd49-ee4665ce8627\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"
Sep 10 23:02:44.392576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38ad9cd3-c937-4c4a-bd49-ee4665ce8627-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-p6sdw\" (UID: \"38ad9cd3-c937-4c4a-bd49-ee4665ce8627\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"
Sep 10 23:02:44.392576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f80e44d-eb36-4008-9f71-89627dc96a5e-ca-trust-extracted\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.392576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.392460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f80e44d-eb36-4008-9f71-89627dc96a5e-registry-certificates\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.404254 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.404224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:44.470976 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.470942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:44.493028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.492995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f80e44d-eb36-4008-9f71-89627dc96a5e-installation-pull-secrets\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.493198 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f80e44d-eb36-4008-9f71-89627dc96a5e-registry-tls\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.493198 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f80e44d-eb36-4008-9f71-89627dc96a5e-bound-sa-token\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.493198 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6szxs\" (UniqueName: \"kubernetes.io/projected/8f80e44d-eb36-4008-9f71-89627dc96a5e-kube-api-access-6szxs\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.493198 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4fae47ba-f91a-48dd-8986-2649ff5f023c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ntjzf\" (UID: \"4fae47ba-f91a-48dd-8986-2649ff5f023c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"
Sep 10 23:02:44.493198 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7brdn\" (UniqueName: \"kubernetes.io/projected/6725a188-ca0f-4af0-832f-172a87277628-kube-api-access-7brdn\") pod \"downloads-76f99584f4-t68b8\" (UID: \"6725a188-ca0f-4af0-832f-172a87277628\") " pod="openshift-console/downloads-76f99584f4-t68b8"
Sep 10 23:02:44.493472 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f80e44d-eb36-4008-9f71-89627dc96a5e-trusted-ca\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.493472 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ad9cd3-c937-4c4a-bd49-ee4665ce8627-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p6sdw\" (UID: \"38ad9cd3-c937-4c4a-bd49-ee4665ce8627\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"
Sep 10 23:02:44.493472 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38ad9cd3-c937-4c4a-bd49-ee4665ce8627-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-p6sdw\" (UID: \"38ad9cd3-c937-4c4a-bd49-ee4665ce8627\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"
Sep 10 23:02:44.493472 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f80e44d-eb36-4008-9f71-89627dc96a5e-ca-trust-extracted\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.493472 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f80e44d-eb36-4008-9f71-89627dc96a5e-registry-certificates\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.493472 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.493381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f80e44d-eb36-4008-9f71-89627dc96a5e-image-registry-private-configuration\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.494362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.494321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38ad9cd3-c937-4c4a-bd49-ee4665ce8627-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-p6sdw\" (UID: \"38ad9cd3-c937-4c4a-bd49-ee4665ce8627\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"
Sep 10 23:02:44.494519 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.494498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f80e44d-eb36-4008-9f71-89627dc96a5e-trusted-ca\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.495199 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.495172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f80e44d-eb36-4008-9f71-89627dc96a5e-registry-certificates\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.495284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.495224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f80e44d-eb36-4008-9f71-89627dc96a5e-ca-trust-extracted\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.496286 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.496258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4fae47ba-f91a-48dd-8986-2649ff5f023c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ntjzf\" (UID: \"4fae47ba-f91a-48dd-8986-2649ff5f023c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"
Sep 10 23:02:44.496424 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.496399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f80e44d-eb36-4008-9f71-89627dc96a5e-installation-pull-secrets\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.496491 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.496460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f80e44d-eb36-4008-9f71-89627dc96a5e-registry-tls\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.496491 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.496467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f80e44d-eb36-4008-9f71-89627dc96a5e-image-registry-private-configuration\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.496491 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.496475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ad9cd3-c937-4c4a-bd49-ee4665ce8627-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p6sdw\" (UID: \"38ad9cd3-c937-4c4a-bd49-ee4665ce8627\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"
Sep 10 23:02:44.501182 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.501155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6szxs\" (UniqueName: \"kubernetes.io/projected/8f80e44d-eb36-4008-9f71-89627dc96a5e-kube-api-access-6szxs\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.501386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.501370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f80e44d-eb36-4008-9f71-89627dc96a5e-bound-sa-token\") pod \"image-registry-77d46bf7c-dtlxq\" (UID: \"8f80e44d-eb36-4008-9f71-89627dc96a5e\") " pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.501433 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.501395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7brdn\" (UniqueName: \"kubernetes.io/projected/6725a188-ca0f-4af0-832f-172a87277628-kube-api-access-7brdn\") pod \"downloads-76f99584f4-t68b8\" (UID: \"6725a188-ca0f-4af0-832f-172a87277628\") " pod="openshift-console/downloads-76f99584f4-t68b8"
Sep 10 23:02:44.630758 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.630717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:44.654525 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.654495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"
Sep 10 23:02:44.665956 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.665927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-t68b8"
Sep 10 23:02:44.671041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.670845    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"
Sep 10 23:02:44.908435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.908361    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d46bf7c-dtlxq"]
Sep 10 23:02:44.911126 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:44.911099    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f80e44d_eb36_4008_9f71_89627dc96a5e.slice/crio-3fcbfc49d79129d3f64ebe145d6b499b450975ec099df23317ea0fc39f9e05c6 WatchSource:0}: Error finding container 3fcbfc49d79129d3f64ebe145d6b499b450975ec099df23317ea0fc39f9e05c6: Status 404 returned error can't find the container with id 3fcbfc49d79129d3f64ebe145d6b499b450975ec099df23317ea0fc39f9e05c6
Sep 10 23:02:44.948409 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.948382    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw"]
Sep 10 23:02:44.951760 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:44.951716    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ad9cd3_c937_4c4a_bd49_ee4665ce8627.slice/crio-7852579b012b7ebd817e3ab18b707d048ed18f7168c18fb992f54e3d9773724c WatchSource:0}: Error finding container 7852579b012b7ebd817e3ab18b707d048ed18f7168c18fb992f54e3d9773724c: Status 404 returned error can't find the container with id 7852579b012b7ebd817e3ab18b707d048ed18f7168c18fb992f54e3d9773724c
Sep 10 23:02:44.962616 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.962587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"]
Sep 10 23:02:44.965951 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:44.965923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-t68b8"]
Sep 10 23:02:44.968557 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:44.968514    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fae47ba_f91a_48dd_8986_2649ff5f023c.slice/crio-502490f40993af519fe595ddf43496336e1466bcdd18811992473f0bd016b95f WatchSource:0}: Error finding container 502490f40993af519fe595ddf43496336e1466bcdd18811992473f0bd016b95f: Status 404 returned error can't find the container with id 502490f40993af519fe595ddf43496336e1466bcdd18811992473f0bd016b95f
Sep 10 23:02:44.969730 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:44.969701    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6725a188_ca0f_4af0_832f_172a87277628.slice/crio-df9f4770fdf229f5c398057eb07a67061b77dd5194e3d19944e1638a0025b0ed WatchSource:0}: Error finding container df9f4770fdf229f5c398057eb07a67061b77dd5194e3d19944e1638a0025b0ed: Status 404 returned error can't find the container with id df9f4770fdf229f5c398057eb07a67061b77dd5194e3d19944e1638a0025b0ed
Sep 10 23:02:45.092684 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.092656    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hstjw"
Sep 10 23:02:45.121529 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.121490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq" event={"ID":"8f80e44d-eb36-4008-9f71-89627dc96a5e","Type":"ContainerStarted","Data":"e7bba6cf2c67cbd2d8dc1c2d223ee7d984bd0d944d7b5bb633d9b95daff82be6"}
Sep 10 23:02:45.121719 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.121572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq" event={"ID":"8f80e44d-eb36-4008-9f71-89627dc96a5e","Type":"ContainerStarted","Data":"3fcbfc49d79129d3f64ebe145d6b499b450975ec099df23317ea0fc39f9e05c6"}
Sep 10 23:02:45.121891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.121643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:02:45.122938 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.122913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw" event={"ID":"38ad9cd3-c937-4c4a-bd49-ee4665ce8627","Type":"ContainerStarted","Data":"7852579b012b7ebd817e3ab18b707d048ed18f7168c18fb992f54e3d9773724c"}
Sep 10 23:02:45.124216 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.124187    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf" event={"ID":"4fae47ba-f91a-48dd-8986-2649ff5f023c","Type":"ContainerStarted","Data":"502490f40993af519fe595ddf43496336e1466bcdd18811992473f0bd016b95f"}
Sep 10 23:02:45.125519 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.125496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-t68b8" event={"ID":"6725a188-ca0f-4af0-832f-172a87277628","Type":"ContainerStarted","Data":"df9f4770fdf229f5c398057eb07a67061b77dd5194e3d19944e1638a0025b0ed"}
Sep 10 23:02:45.143868 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.143814    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq" podStartSLOduration=1.143797934 podStartE2EDuration="1.143797934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:45.143141467 +0000 UTC m=+64.844774415" watchObservedRunningTime="2026-09-10 23:02:45.143797934 +0000 UTC m=+64.845430867"
Sep 10 23:02:45.462141 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.462117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:45.870319 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.870287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:45.951863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:45.951833    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:46.019979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.019897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:46.120849 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.120745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:46.615342 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.615309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:46.615759 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.615355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:46.617845 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.617816    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:02:46.618410 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.618374    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:02:46.628725 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.628702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1071ba46-0db7-4471-8c45-33cd7ca6dc8e-original-pull-secret\") pod \"global-pull-secret-syncer-zsct5\" (UID: \"1071ba46-0db7-4471-8c45-33cd7ca6dc8e\") " pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:46.628834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.628702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7b8f65-5f39-4ce7-b892-c70e379f3437-metrics-certs\") pod \"network-metrics-daemon-v26f5\" (UID: \"6b7b8f65-5f39-4ce7-b892-c70e379f3437\") " pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:46.646811 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.646787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zsct5"
Sep 10 23:02:46.654010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.653984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-glltp\""
Sep 10 23:02:46.662399 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.662377    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v26f5"
Sep 10 23:02:46.716262 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.716224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:46.719302 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.719279    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:02:46.729604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.729583    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:02:46.740453 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.740431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87frl\" (UniqueName: \"kubernetes.io/projected/c7efa847-6bea-4a47-a36b-13a52ae0812c-kube-api-access-87frl\") pod \"network-check-target-2dj82\" (UID: \"c7efa847-6bea-4a47-a36b-13a52ae0812c\") " pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:46.942088 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.942061    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcwbm\""
Sep 10 23:02:46.950129 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:46.949929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:47.134392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.134347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf" event={"ID":"4fae47ba-f91a-48dd-8986-2649ff5f023c","Type":"ContainerStarted","Data":"aba533344e9f7f73774c4a5a8ca1e4ea3b704087182cf1683f78226f3b51d1ed"}
Sep 10 23:02:47.134591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.134452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"
Sep 10 23:02:47.136268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.136235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw" event={"ID":"38ad9cd3-c937-4c4a-bd49-ee4665ce8627","Type":"ContainerStarted","Data":"17addebb86a2571d1a8b28628110c3d9c143f212e416d56f937a7ee79fefa8e9"}
Sep 10 23:02:47.140068 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.140044    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf"
Sep 10 23:02:47.151008 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.150971    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntjzf" podStartSLOduration=1.200433194 podStartE2EDuration="3.150957847s" totalImagesPullingTime="1.950524653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:47.150531003 +0000 UTC m=+66.852163946" watchObservedRunningTime="2026-09-10 23:02:47.150957847 +0000 UTC m=+66.852590790"
Sep 10 23:02:47.166109 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.166023    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6sdw" podStartSLOduration=1.1983732009999999 podStartE2EDuration="3.166003989s" totalImagesPullingTime="1.967630788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:47.165052441 +0000 UTC m=+66.866685385" watchObservedRunningTime="2026-09-10 23:02:47.166003989 +0000 UTC m=+66.867636937"
Sep 10 23:02:47.202749 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.202661    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zsct5"]
Sep 10 23:02:47.205723 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.205655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v26f5"]
Sep 10 23:02:47.206089 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:47.206048    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1071ba46_0db7_4471_8c45_33cd7ca6dc8e.slice/crio-66f5d80f0bef91d9d38c52e28720ce16979e0f196ae410c0b677c06f75dbdc1c WatchSource:0}: Error finding container 66f5d80f0bef91d9d38c52e28720ce16979e0f196ae410c0b677c06f75dbdc1c: Status 404 returned error can't find the container with id 66f5d80f0bef91d9d38c52e28720ce16979e0f196ae410c0b677c06f75dbdc1c
Sep 10 23:02:47.208644 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:47.208616    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7b8f65_5f39_4ce7_b892_c70e379f3437.slice/crio-5c3309fa774c8bc350ed7f607e02b2552e3204bb4a4e36751d4c165903d719b8 WatchSource:0}: Error finding container 5c3309fa774c8bc350ed7f607e02b2552e3204bb4a4e36751d4c165903d719b8: Status 404 returned error can't find the container with id 5c3309fa774c8bc350ed7f607e02b2552e3204bb4a4e36751d4c165903d719b8
Sep 10 23:02:47.243705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.243682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2dj82"]
Sep 10 23:02:47.247099 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:47.247073    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7efa847_6bea_4a47_a36b_13a52ae0812c.slice/crio-982b25b363c0a63728b921169fdd38328d1a2964f05b8a347fe488d4dad4c3dc WatchSource:0}: Error finding container 982b25b363c0a63728b921169fdd38328d1a2964f05b8a347fe488d4dad4c3dc: Status 404 returned error can't find the container with id 982b25b363c0a63728b921169fdd38328d1a2964f05b8a347fe488d4dad4c3dc
Sep 10 23:02:47.280380 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.280353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:47.381856 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.381791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:47.387150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.386926    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"]
Sep 10 23:02:47.390019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.389994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.392255 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.392232    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 23:02:47.392456 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.392433    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 23:02:47.392534 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.392492    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 23:02:47.392606 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.392567    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 23:02:47.392606 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.392577    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-z8b76\""
Sep 10 23:02:47.392827 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.392811    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 23:02:47.398594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.398574    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"]
Sep 10 23:02:47.422949 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.422878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.422949 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.422919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsb68\" (UniqueName: \"kubernetes.io/projected/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-kube-api-access-hsb68\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.423105 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.422993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.423105 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.423030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.451679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.451646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:47.524416 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.524382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.524676 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.524428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.524676 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.524511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.524676 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.524530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsb68\" (UniqueName: \"kubernetes.io/projected/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-kube-api-access-hsb68\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.524676 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:47.524633    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 10 23:02:47.524899 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:47.524708    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-prometheus-operator-tls podName:9cd9abb2-9e7b-4a6f-81a2-695305347b2e nodeName:}" failed. No retries permitted until 2026-09-10 23:02:48.024685546 +0000 UTC m=+67.726318470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-8f82g" (UID: "9cd9abb2-9e7b-4a6f-81a2-695305347b2e") : secret "prometheus-operator-tls" not found
Sep 10 23:02:47.525173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.525152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.527403 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.527375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.533972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.533951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsb68\" (UniqueName: \"kubernetes.io/projected/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-kube-api-access-hsb68\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:47.824207 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:47.824098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:48.029200 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.029163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:48.029384 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.029257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:48.037178 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.037139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399acb8d-754f-4775-a5f8-508412508334-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvkj\" (UID: \"399acb8d-754f-4775-a5f8-508412508334\") " pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:48.037771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.037719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cd9abb2-9e7b-4a6f-81a2-695305347b2e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8f82g\" (UID: \"9cd9abb2-9e7b-4a6f-81a2-695305347b2e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:48.142926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.142888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2dj82" event={"ID":"c7efa847-6bea-4a47-a36b-13a52ae0812c","Type":"ContainerStarted","Data":"982b25b363c0a63728b921169fdd38328d1a2964f05b8a347fe488d4dad4c3dc"}
Sep 10 23:02:48.145393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.145365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zsct5" event={"ID":"1071ba46-0db7-4471-8c45-33cd7ca6dc8e","Type":"ContainerStarted","Data":"66f5d80f0bef91d9d38c52e28720ce16979e0f196ae410c0b677c06f75dbdc1c"}
Sep 10 23:02:48.147585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.147562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v26f5" event={"ID":"6b7b8f65-5f39-4ce7-b892-c70e379f3437","Type":"ContainerStarted","Data":"5c3309fa774c8bc350ed7f607e02b2552e3204bb4a4e36751d4c165903d719b8"}
Sep 10 23:02:48.287693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.287667    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w4w8p\""
Sep 10 23:02:48.296781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.296699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvvkj"
Sep 10 23:02:48.301969 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.301946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"
Sep 10 23:02:48.612812 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.612782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8f82g"]
Sep 10 23:02:48.615789 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:48.615712    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvvkj"]
Sep 10 23:02:48.616291 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:48.616189    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd9abb2_9e7b_4a6f_81a2_695305347b2e.slice/crio-8d9e0d4301383dfb28f950f92e1f03b217330c4f6c325f3b9d4a09478eb90740 WatchSource:0}: Error finding container 8d9e0d4301383dfb28f950f92e1f03b217330c4f6c325f3b9d4a09478eb90740: Status 404 returned error can't find the container with id 8d9e0d4301383dfb28f950f92e1f03b217330c4f6c325f3b9d4a09478eb90740
Sep 10 23:02:48.618643 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:02:48.618600    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399acb8d_754f_4775_a5f8_508412508334.slice/crio-dec3cd64b2a5cbf57490233e8961f0d054eea76d905f3aebb32e39d5499d8f52 WatchSource:0}: Error finding container dec3cd64b2a5cbf57490233e8961f0d054eea76d905f3aebb32e39d5499d8f52: Status 404 returned error can't find the container with id dec3cd64b2a5cbf57490233e8961f0d054eea76d905f3aebb32e39d5499d8f52
Sep 10 23:02:49.177401 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:49.177231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvvkj" event={"ID":"399acb8d-754f-4775-a5f8-508412508334","Type":"ContainerStarted","Data":"0ab038a4fc482b024185e81a28c7855c44d9d6c60df39d7ac02364f94b2f06ad"}
Sep 10 23:02:49.177401 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:49.177349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvvkj" event={"ID":"399acb8d-754f-4775-a5f8-508412508334","Type":"ContainerStarted","Data":"dec3cd64b2a5cbf57490233e8961f0d054eea76d905f3aebb32e39d5499d8f52"}
Sep 10 23:02:49.180213 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:49.180179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g" event={"ID":"9cd9abb2-9e7b-4a6f-81a2-695305347b2e","Type":"ContainerStarted","Data":"8d9e0d4301383dfb28f950f92e1f03b217330c4f6c325f3b9d4a09478eb90740"}
Sep 10 23:02:49.184510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:49.184482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v26f5" event={"ID":"6b7b8f65-5f39-4ce7-b892-c70e379f3437","Type":"ContainerStarted","Data":"19bde2bb53b2ece595a093e1a0f93f0308c553df1ec70aa32f89ebc7af274d64"}
Sep 10 23:02:49.184663 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:49.184532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v26f5" event={"ID":"6b7b8f65-5f39-4ce7-b892-c70e379f3437","Type":"ContainerStarted","Data":"e9751dd986565be2f4b4016e502b33c63631ed23b86d912e0d111618d1164c59"}
Sep 10 23:02:49.200555 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:49.200491    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v26f5" podStartSLOduration=67.138482179 podStartE2EDuration="1m8.200470532s" totalImagesPullingTime="1.061988353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:49.199514101 +0000 UTC m=+68.901147047" watchObservedRunningTime="2026-09-10 23:02:49.200470532 +0000 UTC m=+68.902103478"
Sep 10 23:02:49.290146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:49.290069    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:49.343809 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:49.343772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:50.192626 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:50.192492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvvkj" event={"ID":"399acb8d-754f-4775-a5f8-508412508334","Type":"ContainerStarted","Data":"8d857ed5dee1a30c46876046fef7bd2d164a249e203895c74da1d3105293f55a"}
Sep 10 23:02:54.112790 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.112730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:54.212692 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.212655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2dj82" event={"ID":"c7efa847-6bea-4a47-a36b-13a52ae0812c","Type":"ContainerStarted","Data":"43b97aef09648265386e4d032c66d2d3bb9e2eaa0a4c2f870ba718a22d119fa4"}
Sep 10 23:02:54.212883 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.212774    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:02:54.214712 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.214277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zsct5" event={"ID":"1071ba46-0db7-4471-8c45-33cd7ca6dc8e","Type":"ContainerStarted","Data":"3e59089a9cdb539a6a615cd8672e7af7749e1901ac078bf6f3491852b07da1c2"}
Sep 10 23:02:54.215496 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.215459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:54.217183 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.217156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvvkj" event={"ID":"399acb8d-754f-4775-a5f8-508412508334","Type":"ContainerStarted","Data":"eed8ff120eefad6f04100a007f33404295f216043d059dd4974431ef33530245"}
Sep 10 23:02:54.219158 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.219136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g" event={"ID":"9cd9abb2-9e7b-4a6f-81a2-695305347b2e","Type":"ContainerStarted","Data":"ebfbb057093388396e731ff9db7c75e623d17dbe3d90640049c8a01a79b59697"}
Sep 10 23:02:54.219255 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.219180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g" event={"ID":"9cd9abb2-9e7b-4a6f-81a2-695305347b2e","Type":"ContainerStarted","Data":"d2857d0ec571a8388fafc24e3075b0a673e064ff4f4eb7123e65c0a1b276a5b5"}
Sep 10 23:02:54.229998 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.229962    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2dj82" podStartSLOduration=66.729244296 podStartE2EDuration="1m13.229949384s" totalImagesPullingTime="6.500705088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:54.228384295 +0000 UTC m=+73.930017239" watchObservedRunningTime="2026-09-10 23:02:54.229949384 +0000 UTC m=+73.931582350"
Sep 10 23:02:54.245181 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.245136    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8f82g" podStartSLOduration=2.120208174 podStartE2EDuration="7.245123399s" totalImagesPullingTime="5.124915225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:54.244355872 +0000 UTC m=+73.945988820" watchObservedRunningTime="2026-09-10 23:02:54.245123399 +0000 UTC m=+73.946756333"
Sep 10 23:02:54.259124 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.259078    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zsct5" podStartSLOduration=66.707615536 podStartE2EDuration="1m13.259065198s" totalImagesPullingTime="6.551449662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:54.258568166 +0000 UTC m=+73.960201110" watchObservedRunningTime="2026-09-10 23:02:54.259065198 +0000 UTC m=+73.960698140"
Sep 10 23:02:54.281124 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.281083    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wvvkj" podStartSLOduration=33.328235921 podStartE2EDuration="38.281070976s" totalImagesPullingTime="4.952835055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:02:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:54.279619964 +0000 UTC m=+73.981252906" watchObservedRunningTime="2026-09-10 23:02:54.281070976 +0000 UTC m=+73.982703918"
Sep 10 23:02:54.284280 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.284255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:54.298954 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:54.298931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:02:55.475315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.475283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:55.576126 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.576095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:55.698279 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.698245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:55.752872 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.752789    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zgfl2"]
Sep 10 23:02:55.756801 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.756777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.759324 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.758892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ftsws\""
Sep 10 23:02:55.759646 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.759622    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 23:02:55.759803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.759785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 23:02:55.760092 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.760075    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 23:02:55.766996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.766917    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:55.802914 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.802825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-wtmp\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.802914 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.802878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb5p9\" (UniqueName: \"kubernetes.io/projected/f8e3b7ef-9920-4542-84ce-be45a7805f17-kube-api-access-rb5p9\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.803131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.802933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8e3b7ef-9920-4542-84ce-be45a7805f17-metrics-client-ca\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.803131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.802962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.803131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.802990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8e3b7ef-9920-4542-84ce-be45a7805f17-sys\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.803131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.803016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-textfile\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.803131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.803041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-accelerators-collector-config\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.803370 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.803138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8e3b7ef-9920-4542-84ce-be45a7805f17-root\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.803370 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.803191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-tls\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.837557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.837491    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:55.904607 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.904473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8e3b7ef-9920-4542-84ce-be45a7805f17-root\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.904607 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.904575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8e3b7ef-9920-4542-84ce-be45a7805f17-root\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.904823 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:55.904771    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 23:02:55.904874 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:55.904830    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-tls podName:f8e3b7ef-9920-4542-84ce-be45a7805f17 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:56.404809743 +0000 UTC m=+76.106442666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-tls") pod "node-exporter-zgfl2" (UID: "f8e3b7ef-9920-4542-84ce-be45a7805f17") : secret "node-exporter-tls" not found
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.904531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-tls\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.904976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-wtmp\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.905017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb5p9\" (UniqueName: \"kubernetes.io/projected/f8e3b7ef-9920-4542-84ce-be45a7805f17-kube-api-access-rb5p9\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.905060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8e3b7ef-9920-4542-84ce-be45a7805f17-metrics-client-ca\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.905087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.905117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8e3b7ef-9920-4542-84ce-be45a7805f17-sys\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.905145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-textfile\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.905172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-accelerators-collector-config\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.905186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-wtmp\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.905569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.905485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8e3b7ef-9920-4542-84ce-be45a7805f17-sys\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.906143 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.905786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8e3b7ef-9920-4542-84ce-be45a7805f17-metrics-client-ca\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.906476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.906224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-accelerators-collector-config\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.906476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.906440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-textfile\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.917398 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.917368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.919990 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.919968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb5p9\" (UniqueName: \"kubernetes.io/projected/f8e3b7ef-9920-4542-84ce-be45a7805f17-kube-api-access-rb5p9\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:55.982923 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:55.982851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:56.084128 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:56.084009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:56.184452 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:56.184420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:56.284637 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:56.284608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:56.385888 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:56.385855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:56.411147 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:56.411112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-tls\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:56.411314 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:56.411291    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 23:02:56.411387 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:02:56.411369    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-tls podName:f8e3b7ef-9920-4542-84ce-be45a7805f17 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:57.411347525 +0000 UTC m=+77.112980450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-tls") pod "node-exporter-zgfl2" (UID: "f8e3b7ef-9920-4542-84ce-be45a7805f17") : secret "node-exporter-tls" not found
Sep 10 23:02:56.486034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:56.486003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:56.708619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:56.708434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:56.797859 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:56.797803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:56.919297 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:56.919265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:57.019141 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.019066    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:57.421460 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.421420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-tls\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:57.424228 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.424200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8e3b7ef-9920-4542-84ce-be45a7805f17-node-exporter-tls\") pod \"node-exporter-zgfl2\" (UID: \"f8e3b7ef-9920-4542-84ce-be45a7805f17\") " pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:57.572415 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.572384    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zgfl2"
Sep 10 23:02:57.736756 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.736680    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"]
Sep 10 23:02:57.740187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.740163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.745007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.744984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 23:02:57.745121 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.745032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 23:02:57.745121 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.745057    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 23:02:57.745121 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.745069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-njv52\""
Sep 10 23:02:57.751090 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.751071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 23:02:57.751800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.751782    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 23:02:57.752307 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.752289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e4erafb3015sf\""
Sep 10 23:02:57.764748 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.764726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"]
Sep 10 23:02:57.824326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.824291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.824326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.824332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bc0b864-2ec0-4354-9937-fba124f98cf0-metrics-client-ca\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.824571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.824350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-grpc-tls\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.824571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.824419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-tls\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.824571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.824525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.824720 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.824605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.824720 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.824641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sljw\" (UniqueName: \"kubernetes.io/projected/3bc0b864-2ec0-4354-9937-fba124f98cf0-kube-api-access-7sljw\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.824720 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.824671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.925763 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.925722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.925952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.925777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.925952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.925797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sljw\" (UniqueName: \"kubernetes.io/projected/3bc0b864-2ec0-4354-9937-fba124f98cf0-kube-api-access-7sljw\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.925952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.925824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.925952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.925870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.925952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.925902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bc0b864-2ec0-4354-9937-fba124f98cf0-metrics-client-ca\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.925952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.925926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-grpc-tls\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.925952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.925948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-tls\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.927091 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.926802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bc0b864-2ec0-4354-9937-fba124f98cf0-metrics-client-ca\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.929071 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.929027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.929256 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.929236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-tls\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.929761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.929721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-grpc-tls\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.930643 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.930279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.930643 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.930607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.930765 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.930685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bc0b864-2ec0-4354-9937-fba124f98cf0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:57.936613 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:57.936516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sljw\" (UniqueName: \"kubernetes.io/projected/3bc0b864-2ec0-4354-9937-fba124f98cf0-kube-api-access-7sljw\") pod \"thanos-querier-6c5dd6dbd-czd86\" (UID: \"3bc0b864-2ec0-4354-9937-fba124f98cf0\") " pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:58.053787 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:58.053705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:02:58.700901 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:58.700854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:58.793799 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:58.793723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:58.894518 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:58.894487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:58.998214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:58.998126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:59.052131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.052100    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d89d9f877-m4t8p"]
Sep 10 23:02:59.055510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.055484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.057915 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.057896    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 23:02:59.058042 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.057925    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 23:02:59.058042 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.057928    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 23:02:59.058042 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.057903    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c5cxc\""
Sep 10 23:02:59.058397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.058374    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 23:02:59.058664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.058649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 23:02:59.064478 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.063623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d89d9f877-m4t8p"]
Sep 10 23:02:59.064478 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.064031    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 23:02:59.073218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.072440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:59.136762 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.136724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-serving-cert\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.136928 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.136774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-service-ca\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.136928 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.136857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-config\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.137051 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.136950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-oauth-serving-cert\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.137051 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.137005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-oauth-config\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.137051 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.137038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-trusted-ca-bundle\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.137186 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.137064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntjxk\" (UniqueName: \"kubernetes.io/projected/2b56414a-41e2-4826-b5b9-d0318b9e883d-kube-api-access-ntjxk\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.234413 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.234369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:59.237561 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.237521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-serving-cert\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.237705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.237580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-service-ca\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.237705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.237659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-config\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.237818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.237718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-oauth-serving-cert\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.237818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.237776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-oauth-config\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.237818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.237811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-trusted-ca-bundle\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.237912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.237837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntjxk\" (UniqueName: \"kubernetes.io/projected/2b56414a-41e2-4826-b5b9-d0318b9e883d-kube-api-access-ntjxk\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.238324 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.238297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-config\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.238521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.238500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-service-ca\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.238836 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.238818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-oauth-serving-cert\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.239043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.239020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-trusted-ca-bundle\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.240613 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.240592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-oauth-config\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.240803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.240785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-serving-cert\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.244941 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.244919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntjxk\" (UniqueName: \"kubernetes.io/projected/2b56414a-41e2-4826-b5b9-d0318b9e883d-kube-api-access-ntjxk\") pod \"console-6d89d9f877-m4t8p\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") " pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:02:59.368064 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:02:59.368008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:03:00.136330 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.136293    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cf87649cf-7cjdm"]
Sep 10 23:03:00.139407 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.139385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.142411 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.142378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 23:03:00.142924 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.142660    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 23:03:00.142924 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.142681    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1lfbf4nbcf79h\""
Sep 10 23:03:00.142924 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.142745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-229vt\""
Sep 10 23:03:00.142924 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.142758    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 23:03:00.143189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.143157    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 23:03:00.152750 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.152725    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cf87649cf-7cjdm"]
Sep 10 23:03:00.248865 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.248833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/151ea0af-d668-4ea1-aefc-962ac0de568c-audit-log\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.249024 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.248883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/151ea0af-d668-4ea1-aefc-962ac0de568c-metrics-server-audit-profiles\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.249024 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.248913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcsdd\" (UniqueName: \"kubernetes.io/projected/151ea0af-d668-4ea1-aefc-962ac0de568c-kube-api-access-vcsdd\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.249024 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.248943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/151ea0af-d668-4ea1-aefc-962ac0de568c-secret-metrics-server-tls\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.249166 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.249028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151ea0af-d668-4ea1-aefc-962ac0de568c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.249166 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.249077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/151ea0af-d668-4ea1-aefc-962ac0de568c-client-ca-bundle\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.249267 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.249199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/151ea0af-d668-4ea1-aefc-962ac0de568c-secret-metrics-server-client-certs\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.349626 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.349590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151ea0af-d668-4ea1-aefc-962ac0de568c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.349626 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.349631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/151ea0af-d668-4ea1-aefc-962ac0de568c-client-ca-bundle\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.349852 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.349689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/151ea0af-d668-4ea1-aefc-962ac0de568c-secret-metrics-server-client-certs\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.349852 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.349719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/151ea0af-d668-4ea1-aefc-962ac0de568c-audit-log\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.349852 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.349753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/151ea0af-d668-4ea1-aefc-962ac0de568c-metrics-server-audit-profiles\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.349852 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.349776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcsdd\" (UniqueName: \"kubernetes.io/projected/151ea0af-d668-4ea1-aefc-962ac0de568c-kube-api-access-vcsdd\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.349852 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.349801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/151ea0af-d668-4ea1-aefc-962ac0de568c-secret-metrics-server-tls\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.350432 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.350408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151ea0af-d668-4ea1-aefc-962ac0de568c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.350669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.350627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/151ea0af-d668-4ea1-aefc-962ac0de568c-audit-log\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.351220 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.351172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/151ea0af-d668-4ea1-aefc-962ac0de568c-metrics-server-audit-profiles\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.352837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.352815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/151ea0af-d668-4ea1-aefc-962ac0de568c-secret-metrics-server-tls\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.353393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.353368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/151ea0af-d668-4ea1-aefc-962ac0de568c-client-ca-bundle\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.353492 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.353472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/151ea0af-d668-4ea1-aefc-962ac0de568c-secret-metrics-server-client-certs\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.359106 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.359082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcsdd\" (UniqueName: \"kubernetes.io/projected/151ea0af-d668-4ea1-aefc-962ac0de568c-kube-api-access-vcsdd\") pod \"metrics-server-7cf87649cf-7cjdm\" (UID: \"151ea0af-d668-4ea1-aefc-962ac0de568c\") " pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.454224 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.454135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:00.474966 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.474931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:00.549455 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.549416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:00.553774 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.553749    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75567bfc9d-szs5k"]
Sep 10 23:03:00.557220 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.557200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.569363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.569340    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75567bfc9d-szs5k"]
Sep 10 23:03:00.612162 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.612128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:00.652746 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.652719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-service-ca\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.652903 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.652754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2nhz\" (UniqueName: \"kubernetes.io/projected/2df83c89-ebc0-4a63-a73d-3e021d4b446d-kube-api-access-v2nhz\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.652903 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.652790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-trusted-ca-bundle\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.652903 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.652884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-config\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.653114 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.652910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-serving-cert\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.653114 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.652939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-oauth-config\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.653114 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.652967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-oauth-serving-cert\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.753486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.753409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-trusted-ca-bundle\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.753486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.753478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-config\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.753704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.753510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-serving-cert\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.753704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.753535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-oauth-config\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.753704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.753567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-oauth-serving-cert\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.753704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.753611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-service-ca\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.753704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.753629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2nhz\" (UniqueName: \"kubernetes.io/projected/2df83c89-ebc0-4a63-a73d-3e021d4b446d-kube-api-access-v2nhz\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.755644 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.754969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-trusted-ca-bundle\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.755644 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.755242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-config\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.755644 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.755472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-oauth-serving-cert\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.755644 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.755591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-service-ca\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.757671 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.757632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-oauth-config\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.757985 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.757962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-serving-cert\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.760983 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.760966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2nhz\" (UniqueName: \"kubernetes.io/projected/2df83c89-ebc0-4a63-a73d-3e021d4b446d-kube-api-access-v2nhz\") pod \"console-75567bfc9d-szs5k\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") " pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:00.870107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:00.870065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:01.005292 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:01.005209    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:01.123456 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:01.123398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:01.994279 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:01.994154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:03.473208 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:03:03.473158    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e3b7ef_9920_4542_84ce_be45a7805f17.slice/crio-7515f377d6b4d421bef6b3e019231f383b15061771bac607d60216d22598a9a2 WatchSource:0}: Error finding container 7515f377d6b4d421bef6b3e019231f383b15061771bac607d60216d22598a9a2: Status 404 returned error can't find the container with id 7515f377d6b4d421bef6b3e019231f383b15061771bac607d60216d22598a9a2
Sep 10 23:03:03.745944 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:03.745833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d89d9f877-m4t8p"]
Sep 10 23:03:03.747562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:03.747519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"]
Sep 10 23:03:03.757357 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:03.757338    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75567bfc9d-szs5k"]
Sep 10 23:03:03.769813 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:03.769791    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cf87649cf-7cjdm"]
Sep 10 23:03:03.790851 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:03:03.790815    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b56414a_41e2_4826_b5b9_d0318b9e883d.slice/crio-e985a8743b808b69fbdca2ed04fdefd5d7c1b7753b3ecbb66f628bb9bf0e8912 WatchSource:0}: Error finding container e985a8743b808b69fbdca2ed04fdefd5d7c1b7753b3ecbb66f628bb9bf0e8912: Status 404 returned error can't find the container with id e985a8743b808b69fbdca2ed04fdefd5d7c1b7753b3ecbb66f628bb9bf0e8912
Sep 10 23:03:03.791500 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:03:03.791479    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc0b864_2ec0_4354_9937_fba124f98cf0.slice/crio-c79d8a91ccb1b78203d262784bed50943c5a4fb99a9b9c2b815f3136ca658184 WatchSource:0}: Error finding container c79d8a91ccb1b78203d262784bed50943c5a4fb99a9b9c2b815f3136ca658184: Status 404 returned error can't find the container with id c79d8a91ccb1b78203d262784bed50943c5a4fb99a9b9c2b815f3136ca658184
Sep 10 23:03:03.792532 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:03:03.792509    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df83c89_ebc0_4a63_a73d_3e021d4b446d.slice/crio-a1e128d3f8c277836897f91286a1982262a63ee221f18674d65fc74f1a3e5f94 WatchSource:0}: Error finding container a1e128d3f8c277836897f91286a1982262a63ee221f18674d65fc74f1a3e5f94: Status 404 returned error can't find the container with id a1e128d3f8c277836897f91286a1982262a63ee221f18674d65fc74f1a3e5f94
Sep 10 23:03:03.793199 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:03:03.793175    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151ea0af_d668_4ea1_aefc_962ac0de568c.slice/crio-95bb45eec84dcc015a797f5521639d0a5691c52254af2c8a27434bcb93f9f81e WatchSource:0}: Error finding container 95bb45eec84dcc015a797f5521639d0a5691c52254af2c8a27434bcb93f9f81e: Status 404 returned error can't find the container with id 95bb45eec84dcc015a797f5521639d0a5691c52254af2c8a27434bcb93f9f81e
Sep 10 23:03:04.264667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.264596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75567bfc9d-szs5k" event={"ID":"2df83c89-ebc0-4a63-a73d-3e021d4b446d","Type":"ContainerStarted","Data":"a1e128d3f8c277836897f91286a1982262a63ee221f18674d65fc74f1a3e5f94"}
Sep 10 23:03:04.266207 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.266170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d89d9f877-m4t8p" event={"ID":"2b56414a-41e2-4826-b5b9-d0318b9e883d","Type":"ContainerStarted","Data":"e985a8743b808b69fbdca2ed04fdefd5d7c1b7753b3ecbb66f628bb9bf0e8912"}
Sep 10 23:03:04.269334 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.269271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zgfl2" event={"ID":"f8e3b7ef-9920-4542-84ce-be45a7805f17","Type":"ContainerStarted","Data":"7515f377d6b4d421bef6b3e019231f383b15061771bac607d60216d22598a9a2"}
Sep 10 23:03:04.271093 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.271031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm" event={"ID":"151ea0af-d668-4ea1-aefc-962ac0de568c","Type":"ContainerStarted","Data":"95bb45eec84dcc015a797f5521639d0a5691c52254af2c8a27434bcb93f9f81e"}
Sep 10 23:03:04.274428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.274342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86" event={"ID":"3bc0b864-2ec0-4354-9937-fba124f98cf0","Type":"ContainerStarted","Data":"c79d8a91ccb1b78203d262784bed50943c5a4fb99a9b9c2b815f3136ca658184"}
Sep 10 23:03:04.276755 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.276669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-t68b8" event={"ID":"6725a188-ca0f-4af0-832f-172a87277628","Type":"ContainerStarted","Data":"d8b5cd7d14c067e9fd33bbbc08b0dc7f299d64cb7fd871a31748065215ec18ac"}
Sep 10 23:03:04.279567 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.278438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-t68b8"
Sep 10 23:03:04.288031 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.287977    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-t68b8"
Sep 10 23:03:04.298721 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.298474    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-t68b8" podStartSLOduration=1.410953357 podStartE2EDuration="20.298453735s" totalImagesPullingTime="18.887500378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:04.296913866 +0000 UTC m=+83.998546838" watchObservedRunningTime="2026-09-10 23:03:04.298453735 +0000 UTC m=+84.000086680"
Sep 10 23:03:04.727893 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.727773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:04.828980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:04.828947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:05.316985 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.316907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:05.416275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.416141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:05.529941 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.529864    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d89d9f877-m4t8p"]
Sep 10 23:03:05.557213 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.557178    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75fcbcb677-7bck5"]
Sep 10 23:03:05.567953 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.567717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.571831 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.571767    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75fcbcb677-7bck5"]
Sep 10 23:03:05.701493 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.701374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-serving-cert\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.701493 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.701433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-trusted-ca-bundle\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.701493 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.701488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-oauth-config\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.701835 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.701568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-oauth-serving-cert\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.701835 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.701599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68k7p\" (UniqueName: \"kubernetes.io/projected/1566f37c-1f36-4fd9-b1a8-2395502d31b9-kube-api-access-68k7p\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.701835 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.701663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-service-ca\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.701835 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.701701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-config\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.804571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.803249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-service-ca\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.804571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.803306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-config\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.804571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.803373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-serving-cert\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.804571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.803402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-trusted-ca-bundle\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.804571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.803427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-oauth-config\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.804571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.803477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-oauth-serving-cert\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.804571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.803505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68k7p\" (UniqueName: \"kubernetes.io/projected/1566f37c-1f36-4fd9-b1a8-2395502d31b9-kube-api-access-68k7p\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.804571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.804293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-config\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.804571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.804477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-service-ca\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.805422 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.805125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-trusted-ca-bundle\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.805422 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.805197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-oauth-serving-cert\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.819879 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.819828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-oauth-config\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.824221 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.824186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-serving-cert\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.835506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.835458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68k7p\" (UniqueName: \"kubernetes.io/projected/1566f37c-1f36-4fd9-b1a8-2395502d31b9-kube-api-access-68k7p\") pod \"console-75fcbcb677-7bck5\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") " pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:05.885975 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:05.885943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:06.153565 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:06.151438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77d46bf7c-dtlxq"
Sep 10 23:03:06.256955 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:06.256838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75fcbcb677-7bck5"]
Sep 10 23:03:06.308246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:06.306911    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8e3b7ef-9920-4542-84ce-be45a7805f17" containerID="34f957619955912c2b39347d4fff95084fdcb6a4c0cfbe2912430d33c0d1bec9" exitCode=0
Sep 10 23:03:06.308246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:06.308093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zgfl2" event={"ID":"f8e3b7ef-9920-4542-84ce-be45a7805f17","Type":"ContainerDied","Data":"34f957619955912c2b39347d4fff95084fdcb6a4c0cfbe2912430d33c0d1bec9"}
Sep 10 23:03:06.332384 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:06.332327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:06.396860 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:06.396784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:06.499667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:06.499587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:07.865274 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:07.865194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:07.940506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:07.940469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:08.295640 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:03:08.295561    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1566f37c_1f36_4fd9_b1a8_2395502d31b9.slice/crio-84b32aaa32d64143542b6a7c3391ba9a9d8569db6226792a61bd5baebbe83725 WatchSource:0}: Error finding container 84b32aaa32d64143542b6a7c3391ba9a9d8569db6226792a61bd5baebbe83725: Status 404 returned error can't find the container with id 84b32aaa32d64143542b6a7c3391ba9a9d8569db6226792a61bd5baebbe83725
Sep 10 23:03:08.322143 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:08.322104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fcbcb677-7bck5" event={"ID":"1566f37c-1f36-4fd9-b1a8-2395502d31b9","Type":"ContainerStarted","Data":"84b32aaa32d64143542b6a7c3391ba9a9d8569db6226792a61bd5baebbe83725"}
Sep 10 23:03:09.313289 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:09.313239    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv" podUID="11b68472-05eb-4f97-b418-fa51c8b4a4d3" containerName="registry" containerID="cri-o://cbef6073cff89de494be4ae14d132662bb4271751fda01db4de6dfed5d6ac0c6" gracePeriod=30
Sep 10 23:03:10.214591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.214527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:10.314901 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.314872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:10.333606 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.333571    2580 generic.go:356] "Generic (PLEG): container finished" podID="11b68472-05eb-4f97-b418-fa51c8b4a4d3" containerID="cbef6073cff89de494be4ae14d132662bb4271751fda01db4de6dfed5d6ac0c6" exitCode=0
Sep 10 23:03:10.333767 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.333645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv" event={"ID":"11b68472-05eb-4f97-b418-fa51c8b4a4d3","Type":"ContainerDied","Data":"cbef6073cff89de494be4ae14d132662bb4271751fda01db4de6dfed5d6ac0c6"}
Sep 10 23:03:10.662597 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.662559    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:03:10.766861 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.766791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-bound-sa-token\") pod \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") "
Sep 10 23:03:10.766861 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.766838    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11b68472-05eb-4f97-b418-fa51c8b4a4d3-ca-trust-extracted\") pod \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") "
Sep 10 23:03:10.767027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.766895    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-certificates\") pod \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") "
Sep 10 23:03:10.767027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.766931    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-trusted-ca\") pod \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") "
Sep 10 23:03:10.767027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.766952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndjdv\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-kube-api-access-ndjdv\") pod \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") "
Sep 10 23:03:10.767027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.766977    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-image-registry-private-configuration\") pod \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") "
Sep 10 23:03:10.767215 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.767034    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-installation-pull-secrets\") pod \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") "
Sep 10 23:03:10.767215 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.767090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") pod \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\" (UID: \"11b68472-05eb-4f97-b418-fa51c8b4a4d3\") "
Sep 10 23:03:10.767716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.767620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "11b68472-05eb-4f97-b418-fa51c8b4a4d3" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:10.768169 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.768021    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "11b68472-05eb-4f97-b418-fa51c8b4a4d3" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:10.773234 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.773162    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "11b68472-05eb-4f97-b418-fa51c8b4a4d3" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:10.773601 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.773516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "11b68472-05eb-4f97-b418-fa51c8b4a4d3" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:10.773695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.773661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "11b68472-05eb-4f97-b418-fa51c8b4a4d3" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:03:10.773754 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.773729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "11b68472-05eb-4f97-b418-fa51c8b4a4d3" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:03:10.775378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.775302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-kube-api-access-ndjdv" (OuterVolumeSpecName: "kube-api-access-ndjdv") pod "11b68472-05eb-4f97-b418-fa51c8b4a4d3" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3"). InnerVolumeSpecName "kube-api-access-ndjdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:03:10.779313 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.779280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b68472-05eb-4f97-b418-fa51c8b4a4d3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "11b68472-05eb-4f97-b418-fa51c8b4a4d3" (UID: "11b68472-05eb-4f97-b418-fa51c8b4a4d3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:03:10.868480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.868400    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjdv\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-kube-api-access-ndjdv\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:10.868480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.868434    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-image-registry-private-configuration\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:10.868480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.868449    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11b68472-05eb-4f97-b418-fa51c8b4a4d3-installation-pull-secrets\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:10.868480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.868462    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:10.868480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.868471    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11b68472-05eb-4f97-b418-fa51c8b4a4d3-bound-sa-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:10.868480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.868483    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11b68472-05eb-4f97-b418-fa51c8b4a4d3-ca-trust-extracted\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:10.868821 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.868496    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-registry-certificates\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:10.868821 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:10.868510    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11b68472-05eb-4f97-b418-fa51c8b4a4d3-trusted-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:11.342208 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.342170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zgfl2" event={"ID":"f8e3b7ef-9920-4542-84ce-be45a7805f17","Type":"ContainerStarted","Data":"c99a0382e701e78fad4896af3f7436f4b4914635cc0538b66a4eb642eeb0df78"}
Sep 10 23:03:11.342680 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.342273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zgfl2" event={"ID":"f8e3b7ef-9920-4542-84ce-be45a7805f17","Type":"ContainerStarted","Data":"1f2c12fee4039a5358381f16f593c48016d2397d066e40733e5aeec2db2a4ca0"}
Sep 10 23:03:11.344520 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.344495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fcbcb677-7bck5" event={"ID":"1566f37c-1f36-4fd9-b1a8-2395502d31b9","Type":"ContainerStarted","Data":"6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd"}
Sep 10 23:03:11.346612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.346588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm" event={"ID":"151ea0af-d668-4ea1-aefc-962ac0de568c","Type":"ContainerStarted","Data":"422eeba951c27029ec5cf7c1dd024102ccb2d31116a27e3a68f974f3ddae0f06"}
Sep 10 23:03:11.349330 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.349309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86" event={"ID":"3bc0b864-2ec0-4354-9937-fba124f98cf0","Type":"ContainerStarted","Data":"5e86c5dd3b243a1de3ce6fe4e597b6175296d5fdae83451878f234a948a87255"}
Sep 10 23:03:11.349445 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.349378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86" event={"ID":"3bc0b864-2ec0-4354-9937-fba124f98cf0","Type":"ContainerStarted","Data":"a0f89b57a86b748d6770432e9cee273695790f7c455124d87d667782e42a7003"}
Sep 10 23:03:11.349445 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.349421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86" event={"ID":"3bc0b864-2ec0-4354-9937-fba124f98cf0","Type":"ContainerStarted","Data":"19e7e1afbb26b5c795d83a67eeae4107dea3004327b47aeacce0a2848ca69858"}
Sep 10 23:03:11.350812 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.350787    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv"
Sep 10 23:03:11.350907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.350794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-774fbc58bd-lmddv" event={"ID":"11b68472-05eb-4f97-b418-fa51c8b4a4d3","Type":"ContainerDied","Data":"82968fa1e93706eb92d846b5d643fa2091cd855c5d86c097a73efbd782fa5780"}
Sep 10 23:03:11.350971 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.350922    2580 scope.go:117] "RemoveContainer" containerID="cbef6073cff89de494be4ae14d132662bb4271751fda01db4de6dfed5d6ac0c6"
Sep 10 23:03:11.353172 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.353117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75567bfc9d-szs5k" event={"ID":"2df83c89-ebc0-4a63-a73d-3e021d4b446d","Type":"ContainerStarted","Data":"fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5"}
Sep 10 23:03:11.355269 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.355247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d89d9f877-m4t8p" event={"ID":"2b56414a-41e2-4826-b5b9-d0318b9e883d","Type":"ContainerStarted","Data":"22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba"}
Sep 10 23:03:11.364231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.364025    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zgfl2" podStartSLOduration=14.554049943 podStartE2EDuration="16.364006734s" totalImagesPullingTime="1.67412539s" totalInitContainerRuntime="135.831401ms" isStatefulPod=true podCreationTimestamp="2026-09-10 23:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:11.362390177 +0000 UTC m=+91.064023133" watchObservedRunningTime="2026-09-10 23:03:11.364006734 +0000 UTC m=+91.065639679"
Sep 10 23:03:11.380418 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.380368    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75fcbcb677-7bck5" podStartSLOduration=4.021133081 podStartE2EDuration="6.380349886s" totalImagesPullingTime="2.359216805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:11.379798586 +0000 UTC m=+91.081431529" watchObservedRunningTime="2026-09-10 23:03:11.380349886 +0000 UTC m=+91.081982831"
Sep 10 23:03:11.394011 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.393983    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-774fbc58bd-lmddv"]
Sep 10 23:03:11.397430 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.397408    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-774fbc58bd-lmddv"]
Sep 10 23:03:11.415057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.415007    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d89d9f877-m4t8p" podStartSLOduration=5.376345608 podStartE2EDuration="12.41498946s" totalImagesPullingTime="7.038643852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:11.414396893 +0000 UTC m=+91.116029836" watchObservedRunningTime="2026-09-10 23:03:11.41498946 +0000 UTC m=+91.116622405"
Sep 10 23:03:11.450222 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.450160    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm" podStartSLOduration=4.51528065 podStartE2EDuration="11.45014207s" totalImagesPullingTime="6.93486142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:11.448781849 +0000 UTC m=+91.150414797" watchObservedRunningTime="2026-09-10 23:03:11.45014207 +0000 UTC m=+91.151775013"
Sep 10 23:03:11.450485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:11.450456    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75567bfc9d-szs5k" podStartSLOduration=4.403403114 podStartE2EDuration="11.450444012s" totalImagesPullingTime="7.047040898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:11.432819723 +0000 UTC m=+91.134452665" watchObservedRunningTime="2026-09-10 23:03:11.450444012 +0000 UTC m=+91.152076958"
Sep 10 23:03:12.836739 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:12.833835    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11b68472-05eb-4f97-b418-fa51c8b4a4d3" path="/var/lib/kubelet/pods/11b68472-05eb-4f97-b418-fa51c8b4a4d3/volumes"
Sep 10 23:03:13.367636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:13.367605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86" event={"ID":"3bc0b864-2ec0-4354-9937-fba124f98cf0","Type":"ContainerStarted","Data":"a4763e76bdf2c749939df5770d3b2904bfb38333004df030d19cbcb848436fb4"}
Sep 10 23:03:13.367945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:13.367678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86" event={"ID":"3bc0b864-2ec0-4354-9937-fba124f98cf0","Type":"ContainerStarted","Data":"6a8589999800637bd5b2f009c3c0840bba160292e0463d49fdc1b12f854cdcb4"}
Sep 10 23:03:13.367945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:13.367713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86" event={"ID":"3bc0b864-2ec0-4354-9937-fba124f98cf0","Type":"ContainerStarted","Data":"22719e78625de186e6e03f6979940ba0716c59d337338772d4be24e26a9036dd"}
Sep 10 23:03:13.367945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:13.367776    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:03:13.392701 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:13.392652    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86" podStartSLOduration=8.197965322 podStartE2EDuration="16.392634789s" totalImagesPullingTime="8.194669467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:13.391707858 +0000 UTC m=+93.093340803" watchObservedRunningTime="2026-09-10 23:03:13.392634789 +0000 UTC m=+93.094267733"
Sep 10 23:03:15.886440 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:15.886394    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:15.886887 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:15.886456    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:15.891390 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:15.891370    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:16.035769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:16.035740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:16.136831 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:16.136759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:16.383581 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:16.383557    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:03:16.430491 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:16.430407    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75567bfc9d-szs5k"]
Sep 10 23:03:16.544089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:16.544060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:16.597239 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:16.597218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:16.663042 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:16.663012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:17.306885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:17.306844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:17.367144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:17.367114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:19.368630 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:19.368599    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:03:19.377709 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:19.377687    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6c5dd6dbd-czd86"
Sep 10 23:03:19.510827 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:19.510795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:19.611803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:19.611769    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:20.132041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:20.132015    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vmqvh_106e6c90-7df4-4ff0-954d-81ec5c2b397e/serve-healthcheck-canary/0.log"
Sep 10 23:03:20.454705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:20.454626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:20.454705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:20.454701    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:20.870217 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:20.870184    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:23.141040 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:23.141009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:23.246627 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:23.246599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:23.342092 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:23.342063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:25.225533 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:25.225506    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2dj82"
Sep 10 23:03:26.491246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:26.491213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:30.634695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:30.634662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:30.692143 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:30.692113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:31.021326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:31.021246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:31.122840 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:31.122811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:36.385271 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.385211    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d89d9f877-m4t8p" podUID="2b56414a-41e2-4826-b5b9-d0318b9e883d" containerName="console" containerID="cri-o://22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba" gracePeriod=15
Sep 10 23:03:36.668318 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.668280    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:03:36.789267 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789234    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-config\") pod \"2b56414a-41e2-4826-b5b9-d0318b9e883d\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") "
Sep 10 23:03:36.789435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-serving-cert\") pod \"2b56414a-41e2-4826-b5b9-d0318b9e883d\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") "
Sep 10 23:03:36.789435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntjxk\" (UniqueName: \"kubernetes.io/projected/2b56414a-41e2-4826-b5b9-d0318b9e883d-kube-api-access-ntjxk\") pod \"2b56414a-41e2-4826-b5b9-d0318b9e883d\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") "
Sep 10 23:03:36.789435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-oauth-config\") pod \"2b56414a-41e2-4826-b5b9-d0318b9e883d\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") "
Sep 10 23:03:36.789435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789368    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-service-ca\") pod \"2b56414a-41e2-4826-b5b9-d0318b9e883d\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") "
Sep 10 23:03:36.789435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789389    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-oauth-serving-cert\") pod \"2b56414a-41e2-4826-b5b9-d0318b9e883d\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") "
Sep 10 23:03:36.789722 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-trusted-ca-bundle\") pod \"2b56414a-41e2-4826-b5b9-d0318b9e883d\" (UID: \"2b56414a-41e2-4826-b5b9-d0318b9e883d\") "
Sep 10 23:03:36.789830 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-config" (OuterVolumeSpecName: "console-config") pod "2b56414a-41e2-4826-b5b9-d0318b9e883d" (UID: "2b56414a-41e2-4826-b5b9-d0318b9e883d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:36.789905 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b56414a-41e2-4826-b5b9-d0318b9e883d" (UID: "2b56414a-41e2-4826-b5b9-d0318b9e883d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:36.789956 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789884    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b56414a-41e2-4826-b5b9-d0318b9e883d" (UID: "2b56414a-41e2-4826-b5b9-d0318b9e883d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:36.789995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.789952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2b56414a-41e2-4826-b5b9-d0318b9e883d" (UID: "2b56414a-41e2-4826-b5b9-d0318b9e883d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:36.791908 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.791883    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b56414a-41e2-4826-b5b9-d0318b9e883d-kube-api-access-ntjxk" (OuterVolumeSpecName: "kube-api-access-ntjxk") pod "2b56414a-41e2-4826-b5b9-d0318b9e883d" (UID: "2b56414a-41e2-4826-b5b9-d0318b9e883d"). InnerVolumeSpecName "kube-api-access-ntjxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:03:36.792099 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.792081    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b56414a-41e2-4826-b5b9-d0318b9e883d" (UID: "2b56414a-41e2-4826-b5b9-d0318b9e883d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:36.792170 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.792095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b56414a-41e2-4826-b5b9-d0318b9e883d" (UID: "2b56414a-41e2-4826-b5b9-d0318b9e883d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:36.890354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.890280    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-config\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:36.890354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.890316    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-serving-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:36.890354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.890325    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntjxk\" (UniqueName: \"kubernetes.io/projected/2b56414a-41e2-4826-b5b9-d0318b9e883d-kube-api-access-ntjxk\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:36.890354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.890334    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b56414a-41e2-4826-b5b9-d0318b9e883d-console-oauth-config\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:36.890354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.890342    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:36.890354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.890353    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-oauth-serving-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:36.890354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:36.890361    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b56414a-41e2-4826-b5b9-d0318b9e883d-trusted-ca-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:37.464138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:37.464104    2580 generic.go:356] "Generic (PLEG): container finished" podID="2b56414a-41e2-4826-b5b9-d0318b9e883d" containerID="22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba" exitCode=0
Sep 10 23:03:37.464573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:37.464146    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d89d9f877-m4t8p"
Sep 10 23:03:37.464573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:37.464192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d89d9f877-m4t8p" event={"ID":"2b56414a-41e2-4826-b5b9-d0318b9e883d","Type":"ContainerDied","Data":"22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba"}
Sep 10 23:03:37.464573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:37.464238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d89d9f877-m4t8p" event={"ID":"2b56414a-41e2-4826-b5b9-d0318b9e883d","Type":"ContainerDied","Data":"e985a8743b808b69fbdca2ed04fdefd5d7c1b7753b3ecbb66f628bb9bf0e8912"}
Sep 10 23:03:37.464573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:37.464260    2580 scope.go:117] "RemoveContainer" containerID="22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba"
Sep 10 23:03:37.472328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:37.472303    2580 scope.go:117] "RemoveContainer" containerID="22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba"
Sep 10 23:03:37.472617 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:03:37.472589    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba\": container with ID starting with 22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba not found: ID does not exist" containerID="22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba"
Sep 10 23:03:37.472686 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:37.472633    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba"} err="failed to get container status \"22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba\": rpc error: code = NotFound desc = could not find container \"22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba\": container with ID starting with 22eac8ac217143a139c105e761ad95ca198268d27730d141e34df647de0a5dba not found: ID does not exist"
Sep 10 23:03:37.483050 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:37.481772    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d89d9f877-m4t8p"]
Sep 10 23:03:37.484898 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:37.484879    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d89d9f877-m4t8p"]
Sep 10 23:03:38.832620 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:38.832582    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b56414a-41e2-4826-b5b9-d0318b9e883d" path="/var/lib/kubelet/pods/2b56414a-41e2-4826-b5b9-d0318b9e883d/volumes"
Sep 10 23:03:40.460209 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:40.460182    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:40.464122 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:40.464100    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cf87649cf-7cjdm"
Sep 10 23:03:41.449803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.449738    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75567bfc9d-szs5k" podUID="2df83c89-ebc0-4a63-a73d-3e021d4b446d" containerName="console" containerID="cri-o://fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5" gracePeriod=15
Sep 10 23:03:41.724785 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.724726    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:41.835157 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-oauth-serving-cert\") pod \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") "
Sep 10 23:03:41.835157 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-oauth-config\") pod \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") "
Sep 10 23:03:41.835378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835190    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-serving-cert\") pod \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") "
Sep 10 23:03:41.835378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835240    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-service-ca\") pod \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") "
Sep 10 23:03:41.835378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835277    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2nhz\" (UniqueName: \"kubernetes.io/projected/2df83c89-ebc0-4a63-a73d-3e021d4b446d-kube-api-access-v2nhz\") pod \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") "
Sep 10 23:03:41.835378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835306    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-trusted-ca-bundle\") pod \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") "
Sep 10 23:03:41.835378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-config\") pod \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\" (UID: \"2df83c89-ebc0-4a63-a73d-3e021d4b446d\") "
Sep 10 23:03:41.835669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2df83c89-ebc0-4a63-a73d-3e021d4b446d" (UID: "2df83c89-ebc0-4a63-a73d-3e021d4b446d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:41.835831 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-service-ca" (OuterVolumeSpecName: "service-ca") pod "2df83c89-ebc0-4a63-a73d-3e021d4b446d" (UID: "2df83c89-ebc0-4a63-a73d-3e021d4b446d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:41.836133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-config" (OuterVolumeSpecName: "console-config") pod "2df83c89-ebc0-4a63-a73d-3e021d4b446d" (UID: "2df83c89-ebc0-4a63-a73d-3e021d4b446d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:41.836133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.835885    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2df83c89-ebc0-4a63-a73d-3e021d4b446d" (UID: "2df83c89-ebc0-4a63-a73d-3e021d4b446d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:41.837582 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.837555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2df83c89-ebc0-4a63-a73d-3e021d4b446d" (UID: "2df83c89-ebc0-4a63-a73d-3e021d4b446d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:41.837683 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.837631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2df83c89-ebc0-4a63-a73d-3e021d4b446d" (UID: "2df83c89-ebc0-4a63-a73d-3e021d4b446d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:41.837847 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.837829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df83c89-ebc0-4a63-a73d-3e021d4b446d-kube-api-access-v2nhz" (OuterVolumeSpecName: "kube-api-access-v2nhz") pod "2df83c89-ebc0-4a63-a73d-3e021d4b446d" (UID: "2df83c89-ebc0-4a63-a73d-3e021d4b446d"). InnerVolumeSpecName "kube-api-access-v2nhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:03:41.936422 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.936383    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:41.936615 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.936442    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2nhz\" (UniqueName: \"kubernetes.io/projected/2df83c89-ebc0-4a63-a73d-3e021d4b446d-kube-api-access-v2nhz\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:41.936615 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.936456    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-trusted-ca-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:41.936615 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.936470    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-config\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:41.936615 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.936481    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2df83c89-ebc0-4a63-a73d-3e021d4b446d-oauth-serving-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:41.936615 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.936494    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-oauth-config\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:41.936615 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:41.936506    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2df83c89-ebc0-4a63-a73d-3e021d4b446d-console-serving-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:03:42.485667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.485633    2580 generic.go:356] "Generic (PLEG): container finished" podID="2df83c89-ebc0-4a63-a73d-3e021d4b446d" containerID="fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5" exitCode=0
Sep 10 23:03:42.485857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.485675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75567bfc9d-szs5k" event={"ID":"2df83c89-ebc0-4a63-a73d-3e021d4b446d","Type":"ContainerDied","Data":"fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5"}
Sep 10 23:03:42.485857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.485682    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75567bfc9d-szs5k"
Sep 10 23:03:42.485857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.485715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75567bfc9d-szs5k" event={"ID":"2df83c89-ebc0-4a63-a73d-3e021d4b446d","Type":"ContainerDied","Data":"a1e128d3f8c277836897f91286a1982262a63ee221f18674d65fc74f1a3e5f94"}
Sep 10 23:03:42.485857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.485734    2580 scope.go:117] "RemoveContainer" containerID="fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5"
Sep 10 23:03:42.494236 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.494216    2580 scope.go:117] "RemoveContainer" containerID="fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5"
Sep 10 23:03:42.494507 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:03:42.494471    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5\": container with ID starting with fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5 not found: ID does not exist" containerID="fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5"
Sep 10 23:03:42.494618 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.494518    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5"} err="failed to get container status \"fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5\": rpc error: code = NotFound desc = could not find container \"fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5\": container with ID starting with fa73851acde14bc16df7c50d74de371e4a722e9ae47a38dfa1d13e29306852f5 not found: ID does not exist"
Sep 10 23:03:42.504571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.504552    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75567bfc9d-szs5k"]
Sep 10 23:03:42.508090 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.508066    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75567bfc9d-szs5k"]
Sep 10 23:03:42.832332 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:03:42.832250    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2df83c89-ebc0-4a63-a73d-3e021d4b446d" path="/var/lib/kubelet/pods/2df83c89-ebc0-4a63-a73d-3e021d4b446d/volumes"
Sep 10 23:04:07.286318 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:07.286286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:11.846994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:11.846957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:11.905433 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:11.905398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:23.725045 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:23.725002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:23.825840 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:23.825793    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:31.024440 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:31.024408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:31.288778 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:31.288685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:31.367753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:31.367702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:34.754382 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754349    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c484964cb-7n8kl"]
Sep 10 23:04:34.754782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754764    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b56414a-41e2-4826-b5b9-d0318b9e883d" containerName="console"
Sep 10 23:04:34.754821 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754784    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b56414a-41e2-4826-b5b9-d0318b9e883d" containerName="console"
Sep 10 23:04:34.754821 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754797    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df83c89-ebc0-4a63-a73d-3e021d4b446d" containerName="console"
Sep 10 23:04:34.754821 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754802    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df83c89-ebc0-4a63-a73d-3e021d4b446d" containerName="console"
Sep 10 23:04:34.754821 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754819    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b68472-05eb-4f97-b418-fa51c8b4a4d3" containerName="registry"
Sep 10 23:04:34.754945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754825    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b68472-05eb-4f97-b418-fa51c8b4a4d3" containerName="registry"
Sep 10 23:04:34.754945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754870    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11b68472-05eb-4f97-b418-fa51c8b4a4d3" containerName="registry"
Sep 10 23:04:34.754945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754881    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b56414a-41e2-4826-b5b9-d0318b9e883d" containerName="console"
Sep 10 23:04:34.754945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.754887    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df83c89-ebc0-4a63-a73d-3e021d4b446d" containerName="console"
Sep 10 23:04:34.756762 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.756746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.785230 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.785195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c484964cb-7n8kl"]
Sep 10 23:04:34.829687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.829650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:34.853379 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.853343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-serving-cert\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.853379 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.853380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-console-config\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.853640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.853409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr5pt\" (UniqueName: \"kubernetes.io/projected/446d705d-2b22-4de2-9576-bcea82af0097-kube-api-access-xr5pt\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.853640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.853428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-oauth-serving-cert\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.853640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.853482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-service-ca\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.853640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.853523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-trusted-ca-bundle\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.853640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.853618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-oauth-config\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.929992 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.929960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:34.954302 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.954274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-console-config\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.954385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.954316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr5pt\" (UniqueName: \"kubernetes.io/projected/446d705d-2b22-4de2-9576-bcea82af0097-kube-api-access-xr5pt\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.954453 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.954437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-oauth-serving-cert\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.954508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.954480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-service-ca\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.954590 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.954509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-trusted-ca-bundle\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.954851 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.954820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-oauth-config\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.954960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.954875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-serving-cert\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.955017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.954963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-console-config\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.955213 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.955192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-oauth-serving-cert\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.955428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.955404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-service-ca\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.955650 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.955605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-trusted-ca-bundle\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.957442 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.957420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-oauth-config\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.957523 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.957492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-serving-cert\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:34.962403 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:34.962380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr5pt\" (UniqueName: \"kubernetes.io/projected/446d705d-2b22-4de2-9576-bcea82af0097-kube-api-access-xr5pt\") pod \"console-5c484964cb-7n8kl\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") " pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:35.029975 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:35.029902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:35.065496 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:35.065461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:35.334273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:35.334252    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c484964cb-7n8kl"]
Sep 10 23:04:35.336772 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:04:35.336745    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446d705d_2b22_4de2_9576_bcea82af0097.slice/crio-710a4ccb5396e750c706ac221f9b812be91a0268240f5c1e29eb71c666428389 WatchSource:0}: Error finding container 710a4ccb5396e750c706ac221f9b812be91a0268240f5c1e29eb71c666428389: Status 404 returned error can't find the container with id 710a4ccb5396e750c706ac221f9b812be91a0268240f5c1e29eb71c666428389
Sep 10 23:04:35.682318 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:35.682281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c484964cb-7n8kl" event={"ID":"446d705d-2b22-4de2-9576-bcea82af0097","Type":"ContainerStarted","Data":"8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a"}
Sep 10 23:04:35.682523 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:35.682377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c484964cb-7n8kl" event={"ID":"446d705d-2b22-4de2-9576-bcea82af0097","Type":"ContainerStarted","Data":"710a4ccb5396e750c706ac221f9b812be91a0268240f5c1e29eb71c666428389"}
Sep 10 23:04:35.700457 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:35.700409    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c484964cb-7n8kl" podStartSLOduration=1.7003925180000001 podStartE2EDuration="1.700392518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:04:35.698949655 +0000 UTC m=+175.400582601" watchObservedRunningTime="2026-09-10 23:04:35.700392518 +0000 UTC m=+175.402025460"
Sep 10 23:04:35.827774 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:35.827729    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:35.881734 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:35.881699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:45.066158 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:45.066114    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:45.066714 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:45.066180    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:45.075319 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:45.075297    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:45.721939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:45.721909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:04:45.772205 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:45.772164    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75fcbcb677-7bck5"]
Sep 10 23:04:45.872787 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:45.872755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:45.978813 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:45.978732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:46.030405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:46.030373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:50.264356 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:50.264320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:50.316204 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:50.316175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:52.651391 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:52.651362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:52.705333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:52.705296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:59.865196 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:59.865161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:59.965089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:04:59.965061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:00.555040 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.555006    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"]
Sep 10 23:05:00.558083 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.558067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.560261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.560240    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 23:05:00.560386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.560307    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 23:05:00.560848 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.560831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-99rp9\""
Sep 10 23:05:00.569757 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.569736    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"]
Sep 10 23:05:00.674258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.674233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.674258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.674261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfcx\" (UniqueName: \"kubernetes.io/projected/a4e72a48-d663-415f-8823-13662f26705a-kube-api-access-mtfcx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.674443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.674311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.774820 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.774797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.774981 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.774835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.774981 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.774853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfcx\" (UniqueName: \"kubernetes.io/projected/a4e72a48-d663-415f-8823-13662f26705a-kube-api-access-mtfcx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.775155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.775135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.775224 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.775194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.783167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.783141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfcx\" (UniqueName: \"kubernetes.io/projected/a4e72a48-d663-415f-8823-13662f26705a-kube-api-access-mtfcx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:00.867185 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:00.867159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:01.137161 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:01.137087    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"]
Sep 10 23:05:01.140662 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:05:01.140634    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e72a48_d663_415f_8823_13662f26705a.slice/crio-78991d70ff852b0b221bad4a6c3df7ec9c8de27342fd9e5c6b0aa690d1996f46 WatchSource:0}: Error finding container 78991d70ff852b0b221bad4a6c3df7ec9c8de27342fd9e5c6b0aa690d1996f46: Status 404 returned error can't find the container with id 78991d70ff852b0b221bad4a6c3df7ec9c8de27342fd9e5c6b0aa690d1996f46
Sep 10 23:05:01.777557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:01.777513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs" event={"ID":"a4e72a48-d663-415f-8823-13662f26705a","Type":"ContainerStarted","Data":"78991d70ff852b0b221bad4a6c3df7ec9c8de27342fd9e5c6b0aa690d1996f46"}
Sep 10 23:05:07.806561 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:07.806505    2580 generic.go:356] "Generic (PLEG): container finished" podID="a4e72a48-d663-415f-8823-13662f26705a" containerID="24d7acda74fed9d00c694c98d3e381f1b61ea25863e166260e37b48f4049725b" exitCode=0
Sep 10 23:05:07.807010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:07.806573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs" event={"ID":"a4e72a48-d663-415f-8823-13662f26705a","Type":"ContainerDied","Data":"24d7acda74fed9d00c694c98d3e381f1b61ea25863e166260e37b48f4049725b"}
Sep 10 23:05:10.797104 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:10.796999    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75fcbcb677-7bck5" podUID="1566f37c-1f36-4fd9-b1a8-2395502d31b9" containerName="console" containerID="cri-o://6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd" gracePeriod=15
Sep 10 23:05:10.820455 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:10.820429    2580 generic.go:356] "Generic (PLEG): container finished" podID="a4e72a48-d663-415f-8823-13662f26705a" containerID="44b3acb53f7c65754c87f9781b7b1d7304e328b8e5e48beaa6d20d9c6a3f7995" exitCode=0
Sep 10 23:05:10.820594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:10.820505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs" event={"ID":"a4e72a48-d663-415f-8823-13662f26705a","Type":"ContainerDied","Data":"44b3acb53f7c65754c87f9781b7b1d7304e328b8e5e48beaa6d20d9c6a3f7995"}
Sep 10 23:05:11.035837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.035813    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:05:11.060486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.060416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-config\") pod \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") "
Sep 10 23:05:11.060486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.060465    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-oauth-serving-cert\") pod \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") "
Sep 10 23:05:11.060679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.060500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-service-ca\") pod \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") "
Sep 10 23:05:11.060679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.060533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-trusted-ca-bundle\") pod \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") "
Sep 10 23:05:11.060679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.060578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68k7p\" (UniqueName: \"kubernetes.io/projected/1566f37c-1f36-4fd9-b1a8-2395502d31b9-kube-api-access-68k7p\") pod \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") "
Sep 10 23:05:11.060904 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.060875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-config" (OuterVolumeSpecName: "console-config") pod "1566f37c-1f36-4fd9-b1a8-2395502d31b9" (UID: "1566f37c-1f36-4fd9-b1a8-2395502d31b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:05:11.060904 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.060896    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-service-ca" (OuterVolumeSpecName: "service-ca") pod "1566f37c-1f36-4fd9-b1a8-2395502d31b9" (UID: "1566f37c-1f36-4fd9-b1a8-2395502d31b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:05:11.061022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.060945    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1566f37c-1f36-4fd9-b1a8-2395502d31b9" (UID: "1566f37c-1f36-4fd9-b1a8-2395502d31b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:05:11.061022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.060972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1566f37c-1f36-4fd9-b1a8-2395502d31b9" (UID: "1566f37c-1f36-4fd9-b1a8-2395502d31b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:05:11.062803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.062780    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1566f37c-1f36-4fd9-b1a8-2395502d31b9-kube-api-access-68k7p" (OuterVolumeSpecName: "kube-api-access-68k7p") pod "1566f37c-1f36-4fd9-b1a8-2395502d31b9" (UID: "1566f37c-1f36-4fd9-b1a8-2395502d31b9"). InnerVolumeSpecName "kube-api-access-68k7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:05:11.161770 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.161740    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-oauth-config\") pod \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") "
Sep 10 23:05:11.161931 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.161779    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-serving-cert\") pod \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\" (UID: \"1566f37c-1f36-4fd9-b1a8-2395502d31b9\") "
Sep 10 23:05:11.162009 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.161996    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-config\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:11.162080 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.162016    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-oauth-serving-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:11.162080 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.162031    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:11.162080 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.162042    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1566f37c-1f36-4fd9-b1a8-2395502d31b9-trusted-ca-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:11.162080 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.162054    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68k7p\" (UniqueName: \"kubernetes.io/projected/1566f37c-1f36-4fd9-b1a8-2395502d31b9-kube-api-access-68k7p\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:11.163908 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.163886    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1566f37c-1f36-4fd9-b1a8-2395502d31b9" (UID: "1566f37c-1f36-4fd9-b1a8-2395502d31b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:05:11.163977 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.163932    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1566f37c-1f36-4fd9-b1a8-2395502d31b9" (UID: "1566f37c-1f36-4fd9-b1a8-2395502d31b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:05:11.263170 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.263135    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-oauth-config\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:11.263170 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.263162    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1566f37c-1f36-4fd9-b1a8-2395502d31b9-console-serving-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:11.825494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.825412    2580 generic.go:356] "Generic (PLEG): container finished" podID="1566f37c-1f36-4fd9-b1a8-2395502d31b9" containerID="6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd" exitCode=0
Sep 10 23:05:11.825494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.825463    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fcbcb677-7bck5"
Sep 10 23:05:11.825909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.825504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fcbcb677-7bck5" event={"ID":"1566f37c-1f36-4fd9-b1a8-2395502d31b9","Type":"ContainerDied","Data":"6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd"}
Sep 10 23:05:11.825909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.825575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fcbcb677-7bck5" event={"ID":"1566f37c-1f36-4fd9-b1a8-2395502d31b9","Type":"ContainerDied","Data":"84b32aaa32d64143542b6a7c3391ba9a9d8569db6226792a61bd5baebbe83725"}
Sep 10 23:05:11.825909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.825599    2580 scope.go:117] "RemoveContainer" containerID="6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd"
Sep 10 23:05:11.833875 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.833856    2580 scope.go:117] "RemoveContainer" containerID="6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd"
Sep 10 23:05:11.834150 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:05:11.834133    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd\": container with ID starting with 6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd not found: ID does not exist" containerID="6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd"
Sep 10 23:05:11.834191 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.834159    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd"} err="failed to get container status \"6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd\": rpc error: code = NotFound desc = could not find container \"6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd\": container with ID starting with 6b6973840846a1e1c3793b4ed90a4604ce887a41d2aefddf20237b638531a8cd not found: ID does not exist"
Sep 10 23:05:11.844663 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.844640    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75fcbcb677-7bck5"]
Sep 10 23:05:11.849359 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:11.849337    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75fcbcb677-7bck5"]
Sep 10 23:05:12.833263 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:12.833225    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1566f37c-1f36-4fd9-b1a8-2395502d31b9" path="/var/lib/kubelet/pods/1566f37c-1f36-4fd9-b1a8-2395502d31b9/volumes"
Sep 10 23:05:17.853263 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:17.853227    2580 generic.go:356] "Generic (PLEG): container finished" podID="a4e72a48-d663-415f-8823-13662f26705a" containerID="b985e7477a8cd1c9315a13978cea8d6b6c27b980f6f47c4ead93ddd2a2d3efe8" exitCode=0
Sep 10 23:05:17.853672 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:17.853270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs" event={"ID":"a4e72a48-d663-415f-8823-13662f26705a","Type":"ContainerDied","Data":"b985e7477a8cd1c9315a13978cea8d6b6c27b980f6f47c4ead93ddd2a2d3efe8"}
Sep 10 23:05:18.982435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:18.982405    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:19.027747 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.027717    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtfcx\" (UniqueName: \"kubernetes.io/projected/a4e72a48-d663-415f-8823-13662f26705a-kube-api-access-mtfcx\") pod \"a4e72a48-d663-415f-8823-13662f26705a\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") "
Sep 10 23:05:19.027943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.027770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-util\") pod \"a4e72a48-d663-415f-8823-13662f26705a\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") "
Sep 10 23:05:19.027943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.027794    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-bundle\") pod \"a4e72a48-d663-415f-8823-13662f26705a\" (UID: \"a4e72a48-d663-415f-8823-13662f26705a\") "
Sep 10 23:05:19.028440 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.028369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-bundle" (OuterVolumeSpecName: "bundle") pod "a4e72a48-d663-415f-8823-13662f26705a" (UID: "a4e72a48-d663-415f-8823-13662f26705a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:05:19.030038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.030006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e72a48-d663-415f-8823-13662f26705a-kube-api-access-mtfcx" (OuterVolumeSpecName: "kube-api-access-mtfcx") pod "a4e72a48-d663-415f-8823-13662f26705a" (UID: "a4e72a48-d663-415f-8823-13662f26705a"). InnerVolumeSpecName "kube-api-access-mtfcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:05:19.032551 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.032515    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-util" (OuterVolumeSpecName: "util") pod "a4e72a48-d663-415f-8823-13662f26705a" (UID: "a4e72a48-d663-415f-8823-13662f26705a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:05:19.128909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.128830    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-util\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:19.128909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.128856    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e72a48-d663-415f-8823-13662f26705a-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:19.128909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.128866    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfcx\" (UniqueName: \"kubernetes.io/projected/a4e72a48-d663-415f-8823-13662f26705a-kube-api-access-mtfcx\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:19.862800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.862769    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs"
Sep 10 23:05:19.862933 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.862768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kshxs" event={"ID":"a4e72a48-d663-415f-8823-13662f26705a","Type":"ContainerDied","Data":"78991d70ff852b0b221bad4a6c3df7ec9c8de27342fd9e5c6b0aa690d1996f46"}
Sep 10 23:05:19.862933 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:19.862896    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78991d70ff852b0b221bad4a6c3df7ec9c8de27342fd9e5c6b0aa690d1996f46"
Sep 10 23:05:20.418905 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:20.418853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:20.518439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:20.518407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:21.343525 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:21.343483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:22.722079 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:22.722041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:23.272986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.272957    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"]
Sep 10 23:05:23.273237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273226    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1566f37c-1f36-4fd9-b1a8-2395502d31b9" containerName="console"
Sep 10 23:05:23.273275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273238    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1566f37c-1f36-4fd9-b1a8-2395502d31b9" containerName="console"
Sep 10 23:05:23.273275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273247    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e72a48-d663-415f-8823-13662f26705a" containerName="pull"
Sep 10 23:05:23.273275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273252    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e72a48-d663-415f-8823-13662f26705a" containerName="pull"
Sep 10 23:05:23.273275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273263    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e72a48-d663-415f-8823-13662f26705a" containerName="extract"
Sep 10 23:05:23.273275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273269    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e72a48-d663-415f-8823-13662f26705a" containerName="extract"
Sep 10 23:05:23.273450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273277    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e72a48-d663-415f-8823-13662f26705a" containerName="util"
Sep 10 23:05:23.273450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273282    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e72a48-d663-415f-8823-13662f26705a" containerName="util"
Sep 10 23:05:23.273450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273329    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1566f37c-1f36-4fd9-b1a8-2395502d31b9" containerName="console"
Sep 10 23:05:23.273450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.273360    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e72a48-d663-415f-8823-13662f26705a" containerName="extract"
Sep 10 23:05:23.325876 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.325844    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"]
Sep 10 23:05:23.326008 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.325938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"
Sep 10 23:05:23.328484 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.328462    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 10 23:05:23.328622 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.328462    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:05:23.328622 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.328503    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lw6zf\""
Sep 10 23:05:23.359980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.359956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9pgc\" (UniqueName: \"kubernetes.io/projected/3960fcc7-a7aa-43d4-a5bc-bba587429f0a-kube-api-access-q9pgc\") pod \"cert-manager-operator-controller-manager-55c444bf86-kvs8f\" (UID: \"3960fcc7-a7aa-43d4-a5bc-bba587429f0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"
Sep 10 23:05:23.360109 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.359993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3960fcc7-a7aa-43d4-a5bc-bba587429f0a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kvs8f\" (UID: \"3960fcc7-a7aa-43d4-a5bc-bba587429f0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"
Sep 10 23:05:23.461321 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.461274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9pgc\" (UniqueName: \"kubernetes.io/projected/3960fcc7-a7aa-43d4-a5bc-bba587429f0a-kube-api-access-q9pgc\") pod \"cert-manager-operator-controller-manager-55c444bf86-kvs8f\" (UID: \"3960fcc7-a7aa-43d4-a5bc-bba587429f0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"
Sep 10 23:05:23.461321 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.461327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3960fcc7-a7aa-43d4-a5bc-bba587429f0a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kvs8f\" (UID: \"3960fcc7-a7aa-43d4-a5bc-bba587429f0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"
Sep 10 23:05:23.461719 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.461703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3960fcc7-a7aa-43d4-a5bc-bba587429f0a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kvs8f\" (UID: \"3960fcc7-a7aa-43d4-a5bc-bba587429f0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"
Sep 10 23:05:23.469052 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.469032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9pgc\" (UniqueName: \"kubernetes.io/projected/3960fcc7-a7aa-43d4-a5bc-bba587429f0a-kube-api-access-q9pgc\") pod \"cert-manager-operator-controller-manager-55c444bf86-kvs8f\" (UID: \"3960fcc7-a7aa-43d4-a5bc-bba587429f0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"
Sep 10 23:05:23.634351 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.634320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"
Sep 10 23:05:23.930665 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:23.930638    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f"]
Sep 10 23:05:23.932743 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:05:23.932711    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3960fcc7_a7aa_43d4_a5bc_bba587429f0a.slice/crio-4ae1e46e729df1d75fab491964b5f37e06685cee2ab2edca3b785210db029f62 WatchSource:0}: Error finding container 4ae1e46e729df1d75fab491964b5f37e06685cee2ab2edca3b785210db029f62: Status 404 returned error can't find the container with id 4ae1e46e729df1d75fab491964b5f37e06685cee2ab2edca3b785210db029f62
Sep 10 23:05:24.884206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:24.884170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f" event={"ID":"3960fcc7-a7aa-43d4-a5bc-bba587429f0a","Type":"ContainerStarted","Data":"4ae1e46e729df1d75fab491964b5f37e06685cee2ab2edca3b785210db029f62"}
Sep 10 23:05:24.939336 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:24.939300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:25.035912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:25.035876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:26.895667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:26.895630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f" event={"ID":"3960fcc7-a7aa-43d4-a5bc-bba587429f0a","Type":"ContainerStarted","Data":"6dcd71a4166ef69a315fe44c56ca87d618d33ef28ab24dd605b2ec6258aa0d4e"}
Sep 10 23:05:26.918760 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:26.918696    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kvs8f" podStartSLOduration=1.607822461 podStartE2EDuration="3.91867621s" totalImagesPullingTime="2.310853749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:26.917184178 +0000 UTC m=+226.618817121" watchObservedRunningTime="2026-09-10 23:05:26.91867621 +0000 UTC m=+226.620309154"
Sep 10 23:05:27.115867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.115831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:27.219587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.219474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:27.322866 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.322834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:27.683998 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.683961    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-l8ts9"]
Sep 10 23:05:27.705134 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.705103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-l8ts9"]
Sep 10 23:05:27.705281 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.705191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:27.707679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.707656    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 23:05:27.708347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.708330    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-h29l4\""
Sep 10 23:05:27.708431 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.708337    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 23:05:27.797677 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.797638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2tm\" (UniqueName: \"kubernetes.io/projected/630c6618-deb4-4de3-b605-4b4bba1c465f-kube-api-access-qh2tm\") pod \"cert-manager-webhook-6486ffd997-l8ts9\" (UID: \"630c6618-deb4-4de3-b605-4b4bba1c465f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:27.797847 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.797707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/630c6618-deb4-4de3-b605-4b4bba1c465f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-l8ts9\" (UID: \"630c6618-deb4-4de3-b605-4b4bba1c465f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:27.898627 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.898586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/630c6618-deb4-4de3-b605-4b4bba1c465f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-l8ts9\" (UID: \"630c6618-deb4-4de3-b605-4b4bba1c465f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:27.899110 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.899073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2tm\" (UniqueName: \"kubernetes.io/projected/630c6618-deb4-4de3-b605-4b4bba1c465f-kube-api-access-qh2tm\") pod \"cert-manager-webhook-6486ffd997-l8ts9\" (UID: \"630c6618-deb4-4de3-b605-4b4bba1c465f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:27.906429 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.906398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/630c6618-deb4-4de3-b605-4b4bba1c465f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-l8ts9\" (UID: \"630c6618-deb4-4de3-b605-4b4bba1c465f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:27.906573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.906452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2tm\" (UniqueName: \"kubernetes.io/projected/630c6618-deb4-4de3-b605-4b4bba1c465f-kube-api-access-qh2tm\") pod \"cert-manager-webhook-6486ffd997-l8ts9\" (UID: \"630c6618-deb4-4de3-b605-4b4bba1c465f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:27.962300 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.962202    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"]
Sep 10 23:05:27.976879 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.976852    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"]
Sep 10 23:05:27.977039 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.976953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:27.979266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.979241    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-99rp9\""
Sep 10 23:05:27.979390 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.979292    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 23:05:27.979390 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:27.979315    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 23:05:28.024035 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.024004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:28.101315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.101219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.101315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.101280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.101489 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.101326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr69g\" (UniqueName: \"kubernetes.io/projected/a406f4b3-9684-457f-887b-63ea4f082a74-kube-api-access-hr69g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.202760 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.202721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr69g\" (UniqueName: \"kubernetes.io/projected/a406f4b3-9684-457f-887b-63ea4f082a74-kube-api-access-hr69g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.202932 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.202799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.202932 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.202871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.203221 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.203200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.203260 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.203225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.214600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.213389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr69g\" (UniqueName: \"kubernetes.io/projected/a406f4b3-9684-457f-887b-63ea4f082a74-kube-api-access-hr69g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.286804 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.286769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:28.314660 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.314626    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-l8ts9"]
Sep 10 23:05:28.318315 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:05:28.318282    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod630c6618_deb4_4de3_b605_4b4bba1c465f.slice/crio-aa556637b00281830e6ab161a3a35f1ded9d934c30ffd35ce01bcd7b67053aaf WatchSource:0}: Error finding container aa556637b00281830e6ab161a3a35f1ded9d934c30ffd35ce01bcd7b67053aaf: Status 404 returned error can't find the container with id aa556637b00281830e6ab161a3a35f1ded9d934c30ffd35ce01bcd7b67053aaf
Sep 10 23:05:28.585356 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.585261    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"]
Sep 10 23:05:28.587908 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:05:28.587880    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda406f4b3_9684_457f_887b_63ea4f082a74.slice/crio-e02d61997d0f02b3311238e8c0ad734eaa8284d3f2b26c29d8fb5946e89d676f WatchSource:0}: Error finding container e02d61997d0f02b3311238e8c0ad734eaa8284d3f2b26c29d8fb5946e89d676f: Status 404 returned error can't find the container with id e02d61997d0f02b3311238e8c0ad734eaa8284d3f2b26c29d8fb5946e89d676f
Sep 10 23:05:28.905441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.905408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9" event={"ID":"630c6618-deb4-4de3-b605-4b4bba1c465f","Type":"ContainerStarted","Data":"aa556637b00281830e6ab161a3a35f1ded9d934c30ffd35ce01bcd7b67053aaf"}
Sep 10 23:05:28.906910 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.906882    2580 generic.go:356] "Generic (PLEG): container finished" podID="a406f4b3-9684-457f-887b-63ea4f082a74" containerID="b6f1843575aebd01db4900149b7589f059d34bbd6379caa9b2ef3c51ab6fb415" exitCode=0
Sep 10 23:05:28.907041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.906941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db" event={"ID":"a406f4b3-9684-457f-887b-63ea4f082a74","Type":"ContainerDied","Data":"b6f1843575aebd01db4900149b7589f059d34bbd6379caa9b2ef3c51ab6fb415"}
Sep 10 23:05:28.907041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:28.906989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db" event={"ID":"a406f4b3-9684-457f-887b-63ea4f082a74","Type":"ContainerStarted","Data":"e02d61997d0f02b3311238e8c0ad734eaa8284d3f2b26c29d8fb5946e89d676f"}
Sep 10 23:05:29.516732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:29.516701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:29.694824 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:29.694522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:29.756366 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:29.756332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:31.924715 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:31.924662    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db" event={"ID":"a406f4b3-9684-457f-887b-63ea4f082a74","Type":"ContainerStarted","Data":"f169c74df756eb645cd4836dbe9bac3181a67bb033227a9ee1b91b9a14f50429"}
Sep 10 23:05:31.926810 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:31.926785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9" event={"ID":"630c6618-deb4-4de3-b605-4b4bba1c465f","Type":"ContainerStarted","Data":"a1284bda35eab2bf21cb02452ec0bd836c5c5dc17089bfe505a8b5df7514ccf0"}
Sep 10 23:05:31.926939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:31.926925    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:31.956411 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:31.956349    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9" podStartSLOduration=1.430996709 podStartE2EDuration="4.95632719s" totalImagesPullingTime="3.525330481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:31.954900461 +0000 UTC m=+231.656533407" watchObservedRunningTime="2026-09-10 23:05:31.95632719 +0000 UTC m=+231.657960134"
Sep 10 23:05:32.131206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:32.131171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:32.187866 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:32.187834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:32.937327 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:32.937287    2580 generic.go:356] "Generic (PLEG): container finished" podID="a406f4b3-9684-457f-887b-63ea4f082a74" containerID="f169c74df756eb645cd4836dbe9bac3181a67bb033227a9ee1b91b9a14f50429" exitCode=0
Sep 10 23:05:32.937820 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:32.937374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db" event={"ID":"a406f4b3-9684-457f-887b-63ea4f082a74","Type":"ContainerDied","Data":"f169c74df756eb645cd4836dbe9bac3181a67bb033227a9ee1b91b9a14f50429"}
Sep 10 23:05:33.943987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:33.943952    2580 generic.go:356] "Generic (PLEG): container finished" podID="a406f4b3-9684-457f-887b-63ea4f082a74" containerID="7854eb3bb36db00fef7bcfdb58a0c783985cd67ea68fbd9ebb93703de9518399" exitCode=0
Sep 10 23:05:33.944351 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:33.944009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db" event={"ID":"a406f4b3-9684-457f-887b-63ea4f082a74","Type":"ContainerDied","Data":"7854eb3bb36db00fef7bcfdb58a0c783985cd67ea68fbd9ebb93703de9518399"}
Sep 10 23:05:35.070792 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.070762    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:35.165172 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.165135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-util\") pod \"a406f4b3-9684-457f-887b-63ea4f082a74\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") "
Sep 10 23:05:35.165362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.165215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr69g\" (UniqueName: \"kubernetes.io/projected/a406f4b3-9684-457f-887b-63ea4f082a74-kube-api-access-hr69g\") pod \"a406f4b3-9684-457f-887b-63ea4f082a74\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") "
Sep 10 23:05:35.165362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.165246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-bundle\") pod \"a406f4b3-9684-457f-887b-63ea4f082a74\" (UID: \"a406f4b3-9684-457f-887b-63ea4f082a74\") "
Sep 10 23:05:35.165687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.165652    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-bundle" (OuterVolumeSpecName: "bundle") pod "a406f4b3-9684-457f-887b-63ea4f082a74" (UID: "a406f4b3-9684-457f-887b-63ea4f082a74"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:05:35.167352 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.167332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a406f4b3-9684-457f-887b-63ea4f082a74-kube-api-access-hr69g" (OuterVolumeSpecName: "kube-api-access-hr69g") pod "a406f4b3-9684-457f-887b-63ea4f082a74" (UID: "a406f4b3-9684-457f-887b-63ea4f082a74"). InnerVolumeSpecName "kube-api-access-hr69g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:05:35.170230 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.170195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-util" (OuterVolumeSpecName: "util") pod "a406f4b3-9684-457f-887b-63ea4f082a74" (UID: "a406f4b3-9684-457f-887b-63ea4f082a74"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:05:35.266639 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.266560    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr69g\" (UniqueName: \"kubernetes.io/projected/a406f4b3-9684-457f-887b-63ea4f082a74-kube-api-access-hr69g\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:35.266639 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.266586    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:35.266639 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.266595    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a406f4b3-9684-457f-887b-63ea4f082a74-util\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:35.954532 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.954495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db" event={"ID":"a406f4b3-9684-457f-887b-63ea4f082a74","Type":"ContainerDied","Data":"e02d61997d0f02b3311238e8c0ad734eaa8284d3f2b26c29d8fb5946e89d676f"}
Sep 10 23:05:35.954532 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.954523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvn6db"
Sep 10 23:05:35.954821 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:35.954566    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e02d61997d0f02b3311238e8c0ad734eaa8284d3f2b26c29d8fb5946e89d676f"
Sep 10 23:05:36.723770 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:36.723738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:37.481704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.481669    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8nj6p"]
Sep 10 23:05:37.481971 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.481958    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a406f4b3-9684-457f-887b-63ea4f082a74" containerName="pull"
Sep 10 23:05:37.482015 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.481973    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a406f4b3-9684-457f-887b-63ea4f082a74" containerName="pull"
Sep 10 23:05:37.482015 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.481987    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a406f4b3-9684-457f-887b-63ea4f082a74" containerName="util"
Sep 10 23:05:37.482015 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.481993    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a406f4b3-9684-457f-887b-63ea4f082a74" containerName="util"
Sep 10 23:05:37.482015 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.482002    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a406f4b3-9684-457f-887b-63ea4f082a74" containerName="extract"
Sep 10 23:05:37.482015 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.482007    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a406f4b3-9684-457f-887b-63ea4f082a74" containerName="extract"
Sep 10 23:05:37.482158 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.482058    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a406f4b3-9684-457f-887b-63ea4f082a74" containerName="extract"
Sep 10 23:05:37.484255 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.484240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8nj6p"
Sep 10 23:05:37.486075 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.486059    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-l9gtv\""
Sep 10 23:05:37.494918 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.494898    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8nj6p"]
Sep 10 23:05:37.586863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.586833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bb07708-b225-40d8-b492-2700298f76da-bound-sa-token\") pod \"cert-manager-55b4d954d5-8nj6p\" (UID: \"2bb07708-b225-40d8-b492-2700298f76da\") " pod="cert-manager/cert-manager-55b4d954d5-8nj6p"
Sep 10 23:05:37.586863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.586876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ls5h\" (UniqueName: \"kubernetes.io/projected/2bb07708-b225-40d8-b492-2700298f76da-kube-api-access-8ls5h\") pod \"cert-manager-55b4d954d5-8nj6p\" (UID: \"2bb07708-b225-40d8-b492-2700298f76da\") " pod="cert-manager/cert-manager-55b4d954d5-8nj6p"
Sep 10 23:05:37.687610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.687577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ls5h\" (UniqueName: \"kubernetes.io/projected/2bb07708-b225-40d8-b492-2700298f76da-kube-api-access-8ls5h\") pod \"cert-manager-55b4d954d5-8nj6p\" (UID: \"2bb07708-b225-40d8-b492-2700298f76da\") " pod="cert-manager/cert-manager-55b4d954d5-8nj6p"
Sep 10 23:05:37.687816 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.687667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bb07708-b225-40d8-b492-2700298f76da-bound-sa-token\") pod \"cert-manager-55b4d954d5-8nj6p\" (UID: \"2bb07708-b225-40d8-b492-2700298f76da\") " pod="cert-manager/cert-manager-55b4d954d5-8nj6p"
Sep 10 23:05:37.695459 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.695436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bb07708-b225-40d8-b492-2700298f76da-bound-sa-token\") pod \"cert-manager-55b4d954d5-8nj6p\" (UID: \"2bb07708-b225-40d8-b492-2700298f76da\") " pod="cert-manager/cert-manager-55b4d954d5-8nj6p"
Sep 10 23:05:37.695677 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.695659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ls5h\" (UniqueName: \"kubernetes.io/projected/2bb07708-b225-40d8-b492-2700298f76da-kube-api-access-8ls5h\") pod \"cert-manager-55b4d954d5-8nj6p\" (UID: \"2bb07708-b225-40d8-b492-2700298f76da\") " pod="cert-manager/cert-manager-55b4d954d5-8nj6p"
Sep 10 23:05:37.793411 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.793319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8nj6p"
Sep 10 23:05:37.939909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:37.939883    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-l8ts9"
Sep 10 23:05:38.061349 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:38.061282    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8nj6p"]
Sep 10 23:05:38.064662 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:05:38.064623    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bb07708_b225_40d8_b492_2700298f76da.slice/crio-750772926bb6c4d7aa39b813faa18efdcca1f621ac069941008a48a18f829b2e WatchSource:0}: Error finding container 750772926bb6c4d7aa39b813faa18efdcca1f621ac069941008a48a18f829b2e: Status 404 returned error can't find the container with id 750772926bb6c4d7aa39b813faa18efdcca1f621ac069941008a48a18f829b2e
Sep 10 23:05:38.969960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:38.969930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8nj6p" event={"ID":"2bb07708-b225-40d8-b492-2700298f76da","Type":"ContainerStarted","Data":"e9307833746737f0ff038a8c3b9ec90a06fe133b5ddf44739fc92b2e7212d525"}
Sep 10 23:05:38.969960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:38.969970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8nj6p" event={"ID":"2bb07708-b225-40d8-b492-2700298f76da","Type":"ContainerStarted","Data":"750772926bb6c4d7aa39b813faa18efdcca1f621ac069941008a48a18f829b2e"}
Sep 10 23:05:38.987490 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:38.987441    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-8nj6p" podStartSLOduration=1.987426631 podStartE2EDuration="1.987426631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:38.985782999 +0000 UTC m=+238.687415946" watchObservedRunningTime="2026-09-10 23:05:38.987426631 +0000 UTC m=+238.689059574"
Sep 10 23:05:39.826115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:39.826082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:39.925006 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:39.924977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:41.667136 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.667100    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"]
Sep 10 23:05:41.669834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.669817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"
Sep 10 23:05:41.671922 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.671901    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 23:05:41.672488 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.672470    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:05:41.672532 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.672477    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7zktf\""
Sep 10 23:05:41.681553 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.681518    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"]
Sep 10 23:05:41.819684 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.819645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlgb\" (UniqueName: \"kubernetes.io/projected/b48108d1-fd79-45c1-867f-abdf51f5b24e-kube-api-access-mhlgb\") pod \"openshift-lws-operator-fd8ccff4c-fcgph\" (UID: \"b48108d1-fd79-45c1-867f-abdf51f5b24e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"
Sep 10 23:05:41.819684 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.819687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b48108d1-fd79-45c1-867f-abdf51f5b24e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fcgph\" (UID: \"b48108d1-fd79-45c1-867f-abdf51f5b24e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"
Sep 10 23:05:41.920452 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.920371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b48108d1-fd79-45c1-867f-abdf51f5b24e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fcgph\" (UID: \"b48108d1-fd79-45c1-867f-abdf51f5b24e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"
Sep 10 23:05:41.920635 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.920514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlgb\" (UniqueName: \"kubernetes.io/projected/b48108d1-fd79-45c1-867f-abdf51f5b24e-kube-api-access-mhlgb\") pod \"openshift-lws-operator-fd8ccff4c-fcgph\" (UID: \"b48108d1-fd79-45c1-867f-abdf51f5b24e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"
Sep 10 23:05:41.920825 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.920806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b48108d1-fd79-45c1-867f-abdf51f5b24e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fcgph\" (UID: \"b48108d1-fd79-45c1-867f-abdf51f5b24e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"
Sep 10 23:05:41.933941 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.933915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlgb\" (UniqueName: \"kubernetes.io/projected/b48108d1-fd79-45c1-867f-abdf51f5b24e-kube-api-access-mhlgb\") pod \"openshift-lws-operator-fd8ccff4c-fcgph\" (UID: \"b48108d1-fd79-45c1-867f-abdf51f5b24e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"
Sep 10 23:05:41.978503 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:41.978475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"
Sep 10 23:05:42.253702 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:05:42.253657    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb48108d1_fd79_45c1_867f_abdf51f5b24e.slice/crio-ec7e76052cb4802a4cfb6c962fc511741edfe427932c0751e2096f0e7e94dd79 WatchSource:0}: Error finding container ec7e76052cb4802a4cfb6c962fc511741edfe427932c0751e2096f0e7e94dd79: Status 404 returned error can't find the container with id ec7e76052cb4802a4cfb6c962fc511741edfe427932c0751e2096f0e7e94dd79
Sep 10 23:05:42.256425 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:42.256403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph"]
Sep 10 23:05:42.990865 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:42.990809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph" event={"ID":"b48108d1-fd79-45c1-867f-abdf51f5b24e","Type":"ContainerStarted","Data":"ec7e76052cb4802a4cfb6c962fc511741edfe427932c0751e2096f0e7e94dd79"}
Sep 10 23:05:43.997459 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:43.997421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph" event={"ID":"b48108d1-fd79-45c1-867f-abdf51f5b24e","Type":"ContainerStarted","Data":"840c1667dfe308e12586fcd3a9a6221db4be8de59d620bb6e8a32340edf6f798"}
Sep 10 23:05:44.019426 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:44.019382    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fcgph" podStartSLOduration=1.483953389 podStartE2EDuration="3.019368614s" totalImagesPullingTime="1.535415225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:44.01344504 +0000 UTC m=+243.715077983" watchObservedRunningTime="2026-09-10 23:05:44.019368614 +0000 UTC m=+243.721001556"
Sep 10 23:05:44.718565 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:44.718518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:44.818507 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:44.818473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:46.584066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.584032    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"]
Sep 10 23:05:46.586816 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.586799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.589218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.589198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-99rp9\""
Sep 10 23:05:46.589658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.589628    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 23:05:46.589774 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.589632    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 23:05:46.601133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.601107    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"]
Sep 10 23:05:46.665922 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.665889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.666080 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.665951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72f9q\" (UniqueName: \"kubernetes.io/projected/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-kube-api-access-72f9q\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.666080 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.665987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.767007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.766964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.767196 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.767036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72f9q\" (UniqueName: \"kubernetes.io/projected/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-kube-api-access-72f9q\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.767196 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.767073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.767434 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.767409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.767477 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.767462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.775833 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.775803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72f9q\" (UniqueName: \"kubernetes.io/projected/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-kube-api-access-72f9q\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:46.896394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:46.896352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:47.047889 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:47.047833    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:47.177963 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:47.177894    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"]
Sep 10 23:05:47.181075 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:05:47.181041    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7288be_e8d6_4e0f_aaea_d20e1cd9e5bc.slice/crio-adb496e4ed86efcaf1748594f602f3a1fdcf61b15864d49298919d32bdd0c1ce WatchSource:0}: Error finding container adb496e4ed86efcaf1748594f602f3a1fdcf61b15864d49298919d32bdd0c1ce: Status 404 returned error can't find the container with id adb496e4ed86efcaf1748594f602f3a1fdcf61b15864d49298919d32bdd0c1ce
Sep 10 23:05:48.016432 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:48.016402    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerID="dfc3f3e8b58160e555f35b52cc1db68d90793ac1916fcc6569487976f3ad1aac" exitCode=0
Sep 10 23:05:48.016843 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:48.016457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97" event={"ID":"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc","Type":"ContainerDied","Data":"dfc3f3e8b58160e555f35b52cc1db68d90793ac1916fcc6569487976f3ad1aac"}
Sep 10 23:05:48.016843 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:48.016493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97" event={"ID":"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc","Type":"ContainerStarted","Data":"adb496e4ed86efcaf1748594f602f3a1fdcf61b15864d49298919d32bdd0c1ce"}
Sep 10 23:05:49.022270 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:49.022193    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerID="632943f14423a54a0caf3192611b559f0fb7649e46e4c59b56ef5b8d8dbe8f35" exitCode=0
Sep 10 23:05:49.022685 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:49.022265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97" event={"ID":"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc","Type":"ContainerDied","Data":"632943f14423a54a0caf3192611b559f0fb7649e46e4c59b56ef5b8d8dbe8f35"}
Sep 10 23:05:50.029078 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:50.029041    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerID="2f802f1d10b5478a9c1fe2b5c9b290a8027e9852d66c29dd76417a95e2222d20" exitCode=0
Sep 10 23:05:50.029432 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:50.029130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97" event={"ID":"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc","Type":"ContainerDied","Data":"2f802f1d10b5478a9c1fe2b5c9b290a8027e9852d66c29dd76417a95e2222d20"}
Sep 10 23:05:51.160154 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.160134    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:51.308857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.308781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-util\") pod \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") "
Sep 10 23:05:51.308857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.308824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-bundle\") pod \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") "
Sep 10 23:05:51.308857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.308843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72f9q\" (UniqueName: \"kubernetes.io/projected/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-kube-api-access-72f9q\") pod \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\" (UID: \"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc\") "
Sep 10 23:05:51.309487 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.309458    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-bundle" (OuterVolumeSpecName: "bundle") pod "fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" (UID: "fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:05:51.310938 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.310908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-kube-api-access-72f9q" (OuterVolumeSpecName: "kube-api-access-72f9q") pod "fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" (UID: "fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc"). InnerVolumeSpecName "kube-api-access-72f9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:05:51.314077 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.314043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-util" (OuterVolumeSpecName: "util") pod "fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" (UID: "fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:05:51.410132 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.410094    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-util\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:51.410132 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.410127    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:51.410132 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:51.410136    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72f9q\" (UniqueName: \"kubernetes.io/projected/fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc-kube-api-access-72f9q\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:05:52.039514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:52.039482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97" event={"ID":"fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc","Type":"ContainerDied","Data":"adb496e4ed86efcaf1748594f602f3a1fdcf61b15864d49298919d32bdd0c1ce"}
Sep 10 23:05:52.039694 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:52.039526    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adb496e4ed86efcaf1748594f602f3a1fdcf61b15864d49298919d32bdd0c1ce"
Sep 10 23:05:52.039694 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:52.039568    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af95x97"
Sep 10 23:05:58.581935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:58.581904    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:58.635285 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:05:58.635253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:01.787039 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.786958    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"]
Sep 10 23:06:01.787376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.787249    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerName="extract"
Sep 10 23:06:01.787376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.787260    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerName="extract"
Sep 10 23:06:01.787376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.787276    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerName="util"
Sep 10 23:06:01.787376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.787282    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerName="util"
Sep 10 23:06:01.787376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.787290    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerName="pull"
Sep 10 23:06:01.787376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.787295    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerName="pull"
Sep 10 23:06:01.787376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.787367    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa7288be-e8d6-4e0f-aaea-d20e1cd9e5bc" containerName="extract"
Sep 10 23:06:01.790183 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.790167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:01.792499 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.792479    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 23:06:01.792624 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.792509    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 23:06:01.793241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.793224    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-99rp9\""
Sep 10 23:06:01.795792 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.795774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:01.795876 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.795801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:01.795949 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.795877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqs6\" (UniqueName: \"kubernetes.io/projected/7f88a6e2-7ddb-4443-831c-218941dd0f37-kube-api-access-hwqs6\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:01.799523 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.799503    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"]
Sep 10 23:06:01.896489 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.896450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:01.896704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.896498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:01.896704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.896585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqs6\" (UniqueName: \"kubernetes.io/projected/7f88a6e2-7ddb-4443-831c-218941dd0f37-kube-api-access-hwqs6\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:01.896907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.896886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:01.896969 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.896919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:01.905588 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:01.905564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqs6\" (UniqueName: \"kubernetes.io/projected/7f88a6e2-7ddb-4443-831c-218941dd0f37-kube-api-access-hwqs6\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:02.100073 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:02.100044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:02.369905 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:02.369881    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"]
Sep 10 23:06:02.372066 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:06:02.372042    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f88a6e2_7ddb_4443_831c_218941dd0f37.slice/crio-f91a440e74cfdb598e4c3f9ab3446c3e0a6351b9e0843c859c0904e31445da29 WatchSource:0}: Error finding container f91a440e74cfdb598e4c3f9ab3446c3e0a6351b9e0843c859c0904e31445da29: Status 404 returned error can't find the container with id f91a440e74cfdb598e4c3f9ab3446c3e0a6351b9e0843c859c0904e31445da29
Sep 10 23:06:02.664754 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:02.664676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:02.764699 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:02.764664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:03.090222 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.090132    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerID="e80e581942659e108f533f81c0127c7287ddeb6449f94d6b88ff28b9fb8682b9" exitCode=0
Sep 10 23:06:03.090694 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.090215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt" event={"ID":"7f88a6e2-7ddb-4443-831c-218941dd0f37","Type":"ContainerDied","Data":"e80e581942659e108f533f81c0127c7287ddeb6449f94d6b88ff28b9fb8682b9"}
Sep 10 23:06:03.090694 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.090283    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt" event={"ID":"7f88a6e2-7ddb-4443-831c-218941dd0f37","Type":"ContainerStarted","Data":"f91a440e74cfdb598e4c3f9ab3446c3e0a6351b9e0843c859c0904e31445da29"}
Sep 10 23:06:03.292502 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.292470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:03.385823 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.385794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:03.394564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.394524    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"]
Sep 10 23:06:03.397732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.397717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.400901 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.400883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 23:06:03.401016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.400938    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-m282x\""
Sep 10 23:06:03.401242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.401230    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 23:06:03.401296 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.401230    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 23:06:03.409351 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.409334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtr6\" (UniqueName: \"kubernetes.io/projected/7028e57c-8023-436c-b6cb-56388f386a9c-kube-api-access-gxtr6\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.409466 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.409369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7028e57c-8023-436c-b6cb-56388f386a9c-metrics-cert\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.409466 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.409451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7028e57c-8023-436c-b6cb-56388f386a9c-manager-config\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.409600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.409482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7028e57c-8023-436c-b6cb-56388f386a9c-cert\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.417212 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.417189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"]
Sep 10 23:06:03.510422 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.510389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7028e57c-8023-436c-b6cb-56388f386a9c-manager-config\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.510422 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.510424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7028e57c-8023-436c-b6cb-56388f386a9c-cert\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.510719 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.510480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtr6\" (UniqueName: \"kubernetes.io/projected/7028e57c-8023-436c-b6cb-56388f386a9c-kube-api-access-gxtr6\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.510719 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.510532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7028e57c-8023-436c-b6cb-56388f386a9c-metrics-cert\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.511291 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.511216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7028e57c-8023-436c-b6cb-56388f386a9c-manager-config\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.513126 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.513102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7028e57c-8023-436c-b6cb-56388f386a9c-metrics-cert\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.513225 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.513209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7028e57c-8023-436c-b6cb-56388f386a9c-cert\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.520083 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.520065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtr6\" (UniqueName: \"kubernetes.io/projected/7028e57c-8023-436c-b6cb-56388f386a9c-kube-api-access-gxtr6\") pod \"lws-controller-manager-559dcbdd76-zb5wc\" (UID: \"7028e57c-8023-436c-b6cb-56388f386a9c\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.706389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.706304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:03.982254 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.982223    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:03.982853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:03.982833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"]
Sep 10 23:06:03.998357 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:06:03.998322    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7028e57c_8023_436c_b6cb_56388f386a9c.slice/crio-5717f6d821f584e65a9bd30e198fac382fa48d749996a1760f027a419545996d WatchSource:0}: Error finding container 5717f6d821f584e65a9bd30e198fac382fa48d749996a1760f027a419545996d: Status 404 returned error can't find the container with id 5717f6d821f584e65a9bd30e198fac382fa48d749996a1760f027a419545996d
Sep 10 23:06:04.079411 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.079388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:04.095917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.095893    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerID="621ceeb8c8a6391c24944d974c8e48eda65749b95f30ce706fa79f3a2dd57638" exitCode=0
Sep 10 23:06:04.096249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.095987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt" event={"ID":"7f88a6e2-7ddb-4443-831c-218941dd0f37","Type":"ContainerDied","Data":"621ceeb8c8a6391c24944d974c8e48eda65749b95f30ce706fa79f3a2dd57638"}
Sep 10 23:06:04.097367 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.097289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc" event={"ID":"7028e57c-8023-436c-b6cb-56388f386a9c","Type":"ContainerStarted","Data":"5717f6d821f584e65a9bd30e198fac382fa48d749996a1760f027a419545996d"}
Sep 10 23:06:04.117210 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.117185    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"]
Sep 10 23:06:04.121690 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.121675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.124442 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.124423    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 23:06:04.124867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.124845    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 23:06:04.124970 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.124864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z7b9q\""
Sep 10 23:06:04.125249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.125020    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 23:06:04.127462 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.127440    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 23:06:04.145518 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.145492    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"]
Sep 10 23:06:04.216021 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.215994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/124dfcd6-e648-4a84-8297-27ca3faf1fde-manifests\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.216174 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.216122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/124dfcd6-e648-4a84-8297-27ca3faf1fde-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.216229 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.216171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwn9\" (UniqueName: \"kubernetes.io/projected/124dfcd6-e648-4a84-8297-27ca3faf1fde-kube-api-access-4dwn9\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.216280 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.216226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/124dfcd6-e648-4a84-8297-27ca3faf1fde-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.216280 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.216250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/124dfcd6-e648-4a84-8297-27ca3faf1fde-tmp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.288851 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.288823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:04.316987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.316946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwn9\" (UniqueName: \"kubernetes.io/projected/124dfcd6-e648-4a84-8297-27ca3faf1fde-kube-api-access-4dwn9\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.317163 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.316998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/124dfcd6-e648-4a84-8297-27ca3faf1fde-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.317163 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.317025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/124dfcd6-e648-4a84-8297-27ca3faf1fde-tmp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.317163 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.317067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/124dfcd6-e648-4a84-8297-27ca3faf1fde-manifests\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.317163 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.317130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/124dfcd6-e648-4a84-8297-27ca3faf1fde-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.317602 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.317570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/124dfcd6-e648-4a84-8297-27ca3faf1fde-manifests\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.319434 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.319413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/124dfcd6-e648-4a84-8297-27ca3faf1fde-tmp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.319717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.319699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/124dfcd6-e648-4a84-8297-27ca3faf1fde-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.319772 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.319731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/124dfcd6-e648-4a84-8297-27ca3faf1fde-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.325603 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.325584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwn9\" (UniqueName: \"kubernetes.io/projected/124dfcd6-e648-4a84-8297-27ca3faf1fde-kube-api-access-4dwn9\") pod \"opendatahub-operator-controller-manager-f5b5b6476-227km\" (UID: \"124dfcd6-e648-4a84-8297-27ca3faf1fde\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.389853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.389818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:04.444011 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.443977    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:04.491243 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.491214    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:04.593758 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.593674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:04.693444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.693407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:04.725110 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:04.725011    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"]
Sep 10 23:06:04.728097 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:06:04.728061    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124dfcd6_e648_4a84_8297_27ca3faf1fde.slice/crio-14fc1c9ca1dff3ecd8b5e9568c11eeeaed931f7e1cf6ad40368614c2a4fb37e3 WatchSource:0}: Error finding container 14fc1c9ca1dff3ecd8b5e9568c11eeeaed931f7e1cf6ad40368614c2a4fb37e3: Status 404 returned error can't find the container with id 14fc1c9ca1dff3ecd8b5e9568c11eeeaed931f7e1cf6ad40368614c2a4fb37e3
Sep 10 23:06:05.103258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:05.103225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km" event={"ID":"124dfcd6-e648-4a84-8297-27ca3faf1fde","Type":"ContainerStarted","Data":"14fc1c9ca1dff3ecd8b5e9568c11eeeaed931f7e1cf6ad40368614c2a4fb37e3"}
Sep 10 23:06:05.105788 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:05.105756    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerID="6560009cce9a53cba6ffc62740196edf1fc03131ae54636536bc36ebf76151a9" exitCode=0
Sep 10 23:06:05.105939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:05.105809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt" event={"ID":"7f88a6e2-7ddb-4443-831c-218941dd0f37","Type":"ContainerDied","Data":"6560009cce9a53cba6ffc62740196edf1fc03131ae54636536bc36ebf76151a9"}
Sep 10 23:06:06.114220 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:06.114174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc" event={"ID":"7028e57c-8023-436c-b6cb-56388f386a9c","Type":"ContainerStarted","Data":"53960cec2aeffaa1fa4f7b60bece864b92a4429b184e27e92a42b36bae6d0c59"}
Sep 10 23:06:06.114656 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:06.114265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:06.132805 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:06.132752    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc" podStartSLOduration=1.6513436289999999 podStartE2EDuration="3.132730079s" totalImagesPullingTime="1.48138645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:06.130518087 +0000 UTC m=+265.832151032" watchObservedRunningTime="2026-09-10 23:06:06.132730079 +0000 UTC m=+265.834363026"
Sep 10 23:06:07.931591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:07.931569    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:07.946132 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:07.946111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-util\") pod \"7f88a6e2-7ddb-4443-831c-218941dd0f37\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") "
Sep 10 23:06:07.946235 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:07.946142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-bundle\") pod \"7f88a6e2-7ddb-4443-831c-218941dd0f37\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") "
Sep 10 23:06:07.946235 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:07.946168    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwqs6\" (UniqueName: \"kubernetes.io/projected/7f88a6e2-7ddb-4443-831c-218941dd0f37-kube-api-access-hwqs6\") pod \"7f88a6e2-7ddb-4443-831c-218941dd0f37\" (UID: \"7f88a6e2-7ddb-4443-831c-218941dd0f37\") "
Sep 10 23:06:07.946815 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:07.946791    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-bundle" (OuterVolumeSpecName: "bundle") pod "7f88a6e2-7ddb-4443-831c-218941dd0f37" (UID: "7f88a6e2-7ddb-4443-831c-218941dd0f37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:06:07.948311 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:07.948291    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f88a6e2-7ddb-4443-831c-218941dd0f37-kube-api-access-hwqs6" (OuterVolumeSpecName: "kube-api-access-hwqs6") pod "7f88a6e2-7ddb-4443-831c-218941dd0f37" (UID: "7f88a6e2-7ddb-4443-831c-218941dd0f37"). InnerVolumeSpecName "kube-api-access-hwqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:06:07.952322 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:07.952298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-util" (OuterVolumeSpecName: "util") pod "7f88a6e2-7ddb-4443-831c-218941dd0f37" (UID: "7f88a6e2-7ddb-4443-831c-218941dd0f37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:06:08.047562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:08.047514    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-util\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:06:08.047562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:08.047558    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f88a6e2-7ddb-4443-831c-218941dd0f37-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:06:08.047689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:08.047569    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwqs6\" (UniqueName: \"kubernetes.io/projected/7f88a6e2-7ddb-4443-831c-218941dd0f37-kube-api-access-hwqs6\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:06:08.126885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:08.126857    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt"
Sep 10 23:06:08.127035 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:08.126881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mndgt" event={"ID":"7f88a6e2-7ddb-4443-831c-218941dd0f37","Type":"ContainerDied","Data":"f91a440e74cfdb598e4c3f9ab3446c3e0a6351b9e0843c859c0904e31445da29"}
Sep 10 23:06:08.127035 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:08.126926    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f91a440e74cfdb598e4c3f9ab3446c3e0a6351b9e0843c859c0904e31445da29"
Sep 10 23:06:08.128567 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:08.128525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km" event={"ID":"124dfcd6-e648-4a84-8297-27ca3faf1fde","Type":"ContainerStarted","Data":"7fd6177680fe62909672375ace821fad30ecc8271a352aee0ff364a99db7d7df"}
Sep 10 23:06:09.135344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:09.135317    2580 generic.go:356] "Generic (PLEG): container finished" podID="124dfcd6-e648-4a84-8297-27ca3faf1fde" containerID="7fd6177680fe62909672375ace821fad30ecc8271a352aee0ff364a99db7d7df" exitCode=0
Sep 10 23:06:09.135680 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:09.135395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km" event={"ID":"124dfcd6-e648-4a84-8297-27ca3faf1fde","Type":"ContainerDied","Data":"7fd6177680fe62909672375ace821fad30ecc8271a352aee0ff364a99db7d7df"}
Sep 10 23:06:10.143422 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:10.143384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km" event={"ID":"124dfcd6-e648-4a84-8297-27ca3faf1fde","Type":"ContainerStarted","Data":"a4b4077184ebb11f56865db737009cfb17ba7dc083f3af63001e7565d72e4814"}
Sep 10 23:06:10.143910 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:10.143495    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:10.167271 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:10.167220    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km" podStartSLOduration=1.548353417 podStartE2EDuration="6.167200979s" totalImagesPullingTime="3.526473031s" totalInitContainerRuntime="1.092374531s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:10.165082704 +0000 UTC m=+269.866715648" watchObservedRunningTime="2026-09-10 23:06:10.167200979 +0000 UTC m=+269.868833957"
Sep 10 23:06:14.328105 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:14.328064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:14.428089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:14.428052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:17.122237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:17.122209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-zb5wc"
Sep 10 23:06:21.151093 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:21.151059    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-227km"
Sep 10 23:06:23.794292 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:23.794256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:23.892771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:23.892743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:24.151429 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:24.151391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:24.275198 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:24.275168    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:24.977636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:24.977603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:25.077425 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:25.077383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:25.396583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:25.396521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:25.496466 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:25.496406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:26.425973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:26.425944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:30.724642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:30.724604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:30.824083 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:30.824048    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:31.445977 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.445943    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"]
Sep 10 23:06:31.446373 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.446355    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerName="util"
Sep 10 23:06:31.446476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.446376    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerName="util"
Sep 10 23:06:31.446476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.446398    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerName="extract"
Sep 10 23:06:31.446476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.446406    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerName="extract"
Sep 10 23:06:31.446476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.446426    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerName="pull"
Sep 10 23:06:31.446476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.446436    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerName="pull"
Sep 10 23:06:31.446759 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.446528    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f88a6e2-7ddb-4443-831c-218941dd0f37" containerName="extract"
Sep 10 23:06:31.449626 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.449606    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.451937 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.451915    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 23:06:31.452055 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.451913    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-vtsjb\""
Sep 10 23:06:31.452119 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.451985    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 23:06:31.457613 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.452818    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 23:06:31.465218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.465192    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"]
Sep 10 23:06:31.476773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.476747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:31.546710 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.546676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.546710 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.546716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.546972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.546735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.546972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.546882    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56044d1b-0c10-40ea-8188-e999b0f669b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.546972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.546926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.547098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.546973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.547098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.547002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftwl\" (UniqueName: \"kubernetes.io/projected/56044d1b-0c10-40ea-8188-e999b0f669b8-kube-api-access-fftwl\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.547098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.547074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.547190 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.547117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.648623 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.648583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.648623 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.648623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.648885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.648640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.648885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.648695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56044d1b-0c10-40ea-8188-e999b0f669b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.648885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.648718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.648885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.648762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.648885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.648795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftwl\" (UniqueName: \"kubernetes.io/projected/56044d1b-0c10-40ea-8188-e999b0f669b8-kube-api-access-fftwl\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.648885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.648866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.649196 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.648895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.649248 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.649186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.649248 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.649201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.649491 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.649449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.649640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.649508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.649640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.649624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56044d1b-0c10-40ea-8188-e999b0f669b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.649985 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.649957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:31.651261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.651235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.651580 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.651565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.659150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.659131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56044d1b-0c10-40ea-8188-e999b0f669b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.660275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.660254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftwl\" (UniqueName: \"kubernetes.io/projected/56044d1b-0c10-40ea-8188-e999b0f669b8-kube-api-access-fftwl\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w\" (UID: \"56044d1b-0c10-40ea-8188-e999b0f669b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:31.747973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.747874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:31.766019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:31.765987    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:32.044417 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:32.044347    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"]
Sep 10 23:06:32.048130 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:06:32.048104    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56044d1b_0c10_40ea_8188_e999b0f669b8.slice/crio-11e1a15cccb487d4f2b4f2d6c2bd3139397e2a6fb4d61449a3bca1eb52f52d90 WatchSource:0}: Error finding container 11e1a15cccb487d4f2b4f2d6c2bd3139397e2a6fb4d61449a3bca1eb52f52d90: Status 404 returned error can't find the container with id 11e1a15cccb487d4f2b4f2d6c2bd3139397e2a6fb4d61449a3bca1eb52f52d90
Sep 10 23:06:32.245201 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:32.245157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w" event={"ID":"56044d1b-0c10-40ea-8188-e999b0f669b8","Type":"ContainerStarted","Data":"11e1a15cccb487d4f2b4f2d6c2bd3139397e2a6fb4d61449a3bca1eb52f52d90"}
Sep 10 23:06:34.898400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:34.898366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:35.420341 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:35.420292    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:06:35.420446 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:35.420378    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:06:35.420446 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:35.420412    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:06:35.860067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:35.860031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:36.270188 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:36.270106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w" event={"ID":"56044d1b-0c10-40ea-8188-e999b0f669b8","Type":"ContainerStarted","Data":"88b6147b5768d996926db0e165e340bd57be374ada2824bb6e099ad2be913c22"}
Sep 10 23:06:36.295264 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:36.295210    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w" podStartSLOduration=1.925169305 podStartE2EDuration="5.295191687s" totalImagesPullingTime="3.370022382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:36.294053377 +0000 UTC m=+295.995686322" watchObservedRunningTime="2026-09-10 23:06:36.295191687 +0000 UTC m=+295.996824629"
Sep 10 23:06:36.766977 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:36.766948    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:36.771837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:36.771811    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:37.280847 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:37.280806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:37.281719 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:37.281699    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w"
Sep 10 23:06:40.707515 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:40.707488    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 23:06:44.288689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.288649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:44.346292 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.346262    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4d964"]
Sep 10 23:06:44.348987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.348965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:44.351627 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.351601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 23:06:44.351737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.351683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-x47jw\""
Sep 10 23:06:44.360710 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.360688    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4d964"]
Sep 10 23:06:44.362511 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.362455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:44.461238 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.461209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdcq\" (UniqueName: \"kubernetes.io/projected/5158dc1c-1f72-4bf4-8fe4-7004ceaf4060-kube-api-access-7rdcq\") pod \"kserve-controller-manager-d6f4b65b8-4d964\" (UID: \"5158dc1c-1f72-4bf4-8fe4-7004ceaf4060\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:44.461410 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.461252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5158dc1c-1f72-4bf4-8fe4-7004ceaf4060-cert\") pod \"kserve-controller-manager-d6f4b65b8-4d964\" (UID: \"5158dc1c-1f72-4bf4-8fe4-7004ceaf4060\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:44.562057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.561973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdcq\" (UniqueName: \"kubernetes.io/projected/5158dc1c-1f72-4bf4-8fe4-7004ceaf4060-kube-api-access-7rdcq\") pod \"kserve-controller-manager-d6f4b65b8-4d964\" (UID: \"5158dc1c-1f72-4bf4-8fe4-7004ceaf4060\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:44.562057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.562032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5158dc1c-1f72-4bf4-8fe4-7004ceaf4060-cert\") pod \"kserve-controller-manager-d6f4b65b8-4d964\" (UID: \"5158dc1c-1f72-4bf4-8fe4-7004ceaf4060\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:44.562262 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:06:44.562155    2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 10 23:06:44.562262 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:06:44.562226    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5158dc1c-1f72-4bf4-8fe4-7004ceaf4060-cert podName:5158dc1c-1f72-4bf4-8fe4-7004ceaf4060 nodeName:}" failed. No retries permitted until 2026-09-10 23:06:45.062209695 +0000 UTC m=+304.763842615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5158dc1c-1f72-4bf4-8fe4-7004ceaf4060-cert") pod "kserve-controller-manager-d6f4b65b8-4d964" (UID: "5158dc1c-1f72-4bf4-8fe4-7004ceaf4060") : secret "kserve-webhook-server-cert" not found
Sep 10 23:06:44.572820 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.572791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdcq\" (UniqueName: \"kubernetes.io/projected/5158dc1c-1f72-4bf4-8fe4-7004ceaf4060-kube-api-access-7rdcq\") pod \"kserve-controller-manager-d6f4b65b8-4d964\" (UID: \"5158dc1c-1f72-4bf4-8fe4-7004ceaf4060\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:44.645382 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.645354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:44.752368 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:44.752336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:45.066239 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:45.066200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5158dc1c-1f72-4bf4-8fe4-7004ceaf4060-cert\") pod \"kserve-controller-manager-d6f4b65b8-4d964\" (UID: \"5158dc1c-1f72-4bf4-8fe4-7004ceaf4060\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:45.068718 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:45.068691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5158dc1c-1f72-4bf4-8fe4-7004ceaf4060-cert\") pod \"kserve-controller-manager-d6f4b65b8-4d964\" (UID: \"5158dc1c-1f72-4bf4-8fe4-7004ceaf4060\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:45.263466 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:45.263433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:45.534609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:45.534582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4d964"]
Sep 10 23:06:45.536581 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:06:45.536535    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5158dc1c_1f72_4bf4_8fe4_7004ceaf4060.slice/crio-7666427036e578ab7ff01a673730d8ddcdf1236cf0353091f6fda2b8f54b15c1 WatchSource:0}: Error finding container 7666427036e578ab7ff01a673730d8ddcdf1236cf0353091f6fda2b8f54b15c1: Status 404 returned error can't find the container with id 7666427036e578ab7ff01a673730d8ddcdf1236cf0353091f6fda2b8f54b15c1
Sep 10 23:06:45.537968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:45.537949    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:06:45.564591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:45.564560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:45.624365 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:45.624330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:45.723535 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:45.723499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:46.327682 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:46.327632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964" event={"ID":"5158dc1c-1f72-4bf4-8fe4-7004ceaf4060","Type":"ContainerStarted","Data":"7666427036e578ab7ff01a673730d8ddcdf1236cf0353091f6fda2b8f54b15c1"}
Sep 10 23:06:49.345367 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:49.345334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964" event={"ID":"5158dc1c-1f72-4bf4-8fe4-7004ceaf4060","Type":"ContainerStarted","Data":"d6a81adfbc17b2e2b0b90eaba87d7feba62d546eeb88d136d54f5441b83a1db5"}
Sep 10 23:06:49.345843 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:49.345432    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:06:49.376533 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:06:49.376474    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964" podStartSLOduration=2.589823243 podStartE2EDuration="5.376452319s" totalImagesPullingTime="2.786629076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:49.374149447 +0000 UTC m=+309.075782400" watchObservedRunningTime="2026-09-10 23:06:49.376452319 +0000 UTC m=+309.078085264"
Sep 10 23:07:13.002421 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:13.002371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:20.356530 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:20.356499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4d964"
Sep 10 23:07:56.004679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:56.004621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:56.100420 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:56.100366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:59.465766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.465730    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"]
Sep 10 23:07:59.469271 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.469251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.471403 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.471384    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 23:07:59.471902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.471886    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-99rp9\""
Sep 10 23:07:59.471902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.471892    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 23:07:59.477651 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.477627    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"]
Sep 10 23:07:59.507979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.507952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.508107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.507987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.508107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.508004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn8qh\" (UniqueName: \"kubernetes.io/projected/127fafd2-46e8-486a-b9df-3c48b59ec61b-kube-api-access-sn8qh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.567203 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.567175    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"]
Sep 10 23:07:59.570517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.570499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.579531 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.579508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"]
Sep 10 23:07:59.609406 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.609377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.609513 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.609424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.609513 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.609444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.609513 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.609501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.609688 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.609563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn8qh\" (UniqueName: \"kubernetes.io/projected/127fafd2-46e8-486a-b9df-3c48b59ec61b-kube-api-access-sn8qh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.609688 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.609587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stcsh\" (UniqueName: \"kubernetes.io/projected/51710375-227c-4313-b21e-ee581440728c-kube-api-access-stcsh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.609800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.609783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.609877 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.609858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.618556 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.618522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn8qh\" (UniqueName: \"kubernetes.io/projected/127fafd2-46e8-486a-b9df-3c48b59ec61b-kube-api-access-sn8qh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.668606 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.668578    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"]
Sep 10 23:07:59.672155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.672137    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.681567 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.681528    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"]
Sep 10 23:07:59.710857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.710824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.711019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.710879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.711019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.710909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stcsh\" (UniqueName: \"kubernetes.io/projected/51710375-227c-4313-b21e-ee581440728c-kube-api-access-stcsh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.711019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.710937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7brd\" (UniqueName: \"kubernetes.io/projected/084da37e-406c-4fd6-bb12-1479219bcf7e-kube-api-access-m7brd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.711166 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.711019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.711166 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.711100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.711241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.711191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.711348 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.711330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.721625 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.721573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stcsh\" (UniqueName: \"kubernetes.io/projected/51710375-227c-4313-b21e-ee581440728c-kube-api-access-stcsh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.770977 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.770951    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"]
Sep 10 23:07:59.774535 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.774520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.779782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.779765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:07:59.783723 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.783690    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"]
Sep 10 23:07:59.811814 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.811780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.811957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.811838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.811957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.811946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwls\" (UniqueName: \"kubernetes.io/projected/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-kube-api-access-zvwls\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.812030 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.812013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.812072 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.812039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.812072 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.812061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7brd\" (UniqueName: \"kubernetes.io/projected/084da37e-406c-4fd6-bb12-1479219bcf7e-kube-api-access-m7brd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.812144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.812069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.812438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.812417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.823983 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.823951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7brd\" (UniqueName: \"kubernetes.io/projected/084da37e-406c-4fd6-bb12-1479219bcf7e-kube-api-access-m7brd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:07:59.880133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.880102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:07:59.912779 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.912736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwls\" (UniqueName: \"kubernetes.io/projected/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-kube-api-access-zvwls\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.912779 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.912782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.913007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.912840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.913224 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.913205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.913304 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.913248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.921208 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.921184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwls\" (UniqueName: \"kubernetes.io/projected/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-kube-api-access-zvwls\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:07:59.983555 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:07:59.983463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:08:00.060984 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.060918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"]
Sep 10 23:08:00.084553 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.084513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:08:00.160524 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.160494    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"]
Sep 10 23:08:00.162247 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:08:00.162216    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51710375_227c_4313_b21e_ee581440728c.slice/crio-34bad390488ef329a5b0365e326b16b54e77b783722dd000a9ff54138e5cec89 WatchSource:0}: Error finding container 34bad390488ef329a5b0365e326b16b54e77b783722dd000a9ff54138e5cec89: Status 404 returned error can't find the container with id 34bad390488ef329a5b0365e326b16b54e77b783722dd000a9ff54138e5cec89
Sep 10 23:08:00.258141 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.258119    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"]
Sep 10 23:08:00.259950 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:08:00.259924    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084da37e_406c_4fd6_bb12_1479219bcf7e.slice/crio-3384f59ff8c0b4275b4c30dd803c2dee090e9cfce795e379e9797b1204fb3d53 WatchSource:0}: Error finding container 3384f59ff8c0b4275b4c30dd803c2dee090e9cfce795e379e9797b1204fb3d53: Status 404 returned error can't find the container with id 3384f59ff8c0b4275b4c30dd803c2dee090e9cfce795e379e9797b1204fb3d53
Sep 10 23:08:00.365103 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.365075    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"]
Sep 10 23:08:00.366390 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:08:00.366363    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaefc35d_0580_4f4a_a2e4_e0e9c090fa2f.slice/crio-52826f9da5aee5bc7b720fe3d980affc5e1a1b207a46ab0bd697402af66355fb WatchSource:0}: Error finding container 52826f9da5aee5bc7b720fe3d980affc5e1a1b207a46ab0bd697402af66355fb: Status 404 returned error can't find the container with id 52826f9da5aee5bc7b720fe3d980affc5e1a1b207a46ab0bd697402af66355fb
Sep 10 23:08:00.681319 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.681290    2580 generic.go:356] "Generic (PLEG): container finished" podID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerID="a06b5d1ce5d2d8dc5efe9ad4d91e9735403d80fa27bffc00e09aeeb2b29f4b15" exitCode=0
Sep 10 23:08:00.681711 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.681354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5" event={"ID":"084da37e-406c-4fd6-bb12-1479219bcf7e","Type":"ContainerDied","Data":"a06b5d1ce5d2d8dc5efe9ad4d91e9735403d80fa27bffc00e09aeeb2b29f4b15"}
Sep 10 23:08:00.681711 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.681406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5" event={"ID":"084da37e-406c-4fd6-bb12-1479219bcf7e","Type":"ContainerStarted","Data":"3384f59ff8c0b4275b4c30dd803c2dee090e9cfce795e379e9797b1204fb3d53"}
Sep 10 23:08:00.683046 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.683022    2580 generic.go:356] "Generic (PLEG): container finished" podID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerID="e7b33b86843a8c960f86ec2edb110164bc630d20dda5806ff17f52d6a8ae9db8" exitCode=0
Sep 10 23:08:00.683145 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.683114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh" event={"ID":"127fafd2-46e8-486a-b9df-3c48b59ec61b","Type":"ContainerDied","Data":"e7b33b86843a8c960f86ec2edb110164bc630d20dda5806ff17f52d6a8ae9db8"}
Sep 10 23:08:00.683249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.683168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh" event={"ID":"127fafd2-46e8-486a-b9df-3c48b59ec61b","Type":"ContainerStarted","Data":"a79477d80489a7a3b1f41fdd68d7208b6eca3e229bc853fcc00f0b4b1ccab847"}
Sep 10 23:08:00.684841 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.684820    2580 generic.go:356] "Generic (PLEG): container finished" podID="51710375-227c-4313-b21e-ee581440728c" containerID="2d98f7eea31f0c36b42ddd37d2bbf4412f5aff54af43c7b19ffeaf5fce7f2a8c" exitCode=0
Sep 10 23:08:00.685028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.684934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5" event={"ID":"51710375-227c-4313-b21e-ee581440728c","Type":"ContainerDied","Data":"2d98f7eea31f0c36b42ddd37d2bbf4412f5aff54af43c7b19ffeaf5fce7f2a8c"}
Sep 10 23:08:00.685028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.684975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5" event={"ID":"51710375-227c-4313-b21e-ee581440728c","Type":"ContainerStarted","Data":"34bad390488ef329a5b0365e326b16b54e77b783722dd000a9ff54138e5cec89"}
Sep 10 23:08:00.686529 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.686512    2580 generic.go:356] "Generic (PLEG): container finished" podID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerID="45bc98780487317055b656c3d18ed93f675a044d9118faf78e2171e3da954bdf" exitCode=0
Sep 10 23:08:00.686627 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.686579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw" event={"ID":"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f","Type":"ContainerDied","Data":"45bc98780487317055b656c3d18ed93f675a044d9118faf78e2171e3da954bdf"}
Sep 10 23:08:00.686627 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:00.686610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw" event={"ID":"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f","Type":"ContainerStarted","Data":"52826f9da5aee5bc7b720fe3d980affc5e1a1b207a46ab0bd697402af66355fb"}
Sep 10 23:08:02.700502 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:02.700469    2580 generic.go:356] "Generic (PLEG): container finished" podID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerID="1a5d0a6d7a1c1024c3b5bbb5d3930ca3c66cc356afb07bc968b4d7198edf0b53" exitCode=0
Sep 10 23:08:02.701021 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:02.700569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw" event={"ID":"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f","Type":"ContainerDied","Data":"1a5d0a6d7a1c1024c3b5bbb5d3930ca3c66cc356afb07bc968b4d7198edf0b53"}
Sep 10 23:08:02.702627 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:02.702603    2580 generic.go:356] "Generic (PLEG): container finished" podID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerID="8dbda5373d25bd2726a2576e033cbb981e73d7a469ccf121ec622e3c628a60a2" exitCode=0
Sep 10 23:08:02.702736 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:02.702684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5" event={"ID":"084da37e-406c-4fd6-bb12-1479219bcf7e","Type":"ContainerDied","Data":"8dbda5373d25bd2726a2576e033cbb981e73d7a469ccf121ec622e3c628a60a2"}
Sep 10 23:08:02.704568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:02.704533    2580 generic.go:356] "Generic (PLEG): container finished" podID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerID="c30aff5fc36733aa58fcbebed42ea44b62da9d103dac6b49291259700299c250" exitCode=0
Sep 10 23:08:02.704666 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:02.704626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh" event={"ID":"127fafd2-46e8-486a-b9df-3c48b59ec61b","Type":"ContainerDied","Data":"c30aff5fc36733aa58fcbebed42ea44b62da9d103dac6b49291259700299c250"}
Sep 10 23:08:02.706528 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:02.706511    2580 generic.go:356] "Generic (PLEG): container finished" podID="51710375-227c-4313-b21e-ee581440728c" containerID="9a847b98f4b9cc152348f3c47e8f9b4c7035b1ac8db391c48fdbf7a03a268692" exitCode=0
Sep 10 23:08:02.706641 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:02.706582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5" event={"ID":"51710375-227c-4313-b21e-ee581440728c","Type":"ContainerDied","Data":"9a847b98f4b9cc152348f3c47e8f9b4c7035b1ac8db391c48fdbf7a03a268692"}
Sep 10 23:08:03.713706 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:03.713673    2580 generic.go:356] "Generic (PLEG): container finished" podID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerID="e013c739c1656a4887ac83061ecc84dfe262c26de37990f57e49a6c30163db75" exitCode=0
Sep 10 23:08:03.714069 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:03.713761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw" event={"ID":"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f","Type":"ContainerDied","Data":"e013c739c1656a4887ac83061ecc84dfe262c26de37990f57e49a6c30163db75"}
Sep 10 23:08:03.715870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:03.715848    2580 generic.go:356] "Generic (PLEG): container finished" podID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerID="25b860792e236d24151f4508725ca74d1ed36fb03696a0f736793b78f77c95ea" exitCode=0
Sep 10 23:08:03.715983 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:03.715927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5" event={"ID":"084da37e-406c-4fd6-bb12-1479219bcf7e","Type":"ContainerDied","Data":"25b860792e236d24151f4508725ca74d1ed36fb03696a0f736793b78f77c95ea"}
Sep 10 23:08:03.718098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:03.718081    2580 generic.go:356] "Generic (PLEG): container finished" podID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerID="357f0c202f9952a7ce3a26fcc204cb62fecb5ab90b50dceeba41caf39d78e22e" exitCode=0
Sep 10 23:08:03.718182 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:03.718162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh" event={"ID":"127fafd2-46e8-486a-b9df-3c48b59ec61b","Type":"ContainerDied","Data":"357f0c202f9952a7ce3a26fcc204cb62fecb5ab90b50dceeba41caf39d78e22e"}
Sep 10 23:08:03.720164 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:03.720144    2580 generic.go:356] "Generic (PLEG): container finished" podID="51710375-227c-4313-b21e-ee581440728c" containerID="818f951309d9275d457735f5d83a081eece1256cbb73e18534f6a83f67878208" exitCode=0
Sep 10 23:08:03.720244 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:03.720185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5" event={"ID":"51710375-227c-4313-b21e-ee581440728c","Type":"ContainerDied","Data":"818f951309d9275d457735f5d83a081eece1256cbb73e18534f6a83f67878208"}
Sep 10 23:08:04.887058 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.887026    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:08:04.916005 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.915978    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:08:04.939105 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.939085    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:08:04.941847 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.941827    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:08:04.954767 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.954727    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-util\") pod \"127fafd2-46e8-486a-b9df-3c48b59ec61b\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") "
Sep 10 23:08:04.954767 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.954756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-util\") pod \"51710375-227c-4313-b21e-ee581440728c\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.954798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn8qh\" (UniqueName: \"kubernetes.io/projected/127fafd2-46e8-486a-b9df-3c48b59ec61b-kube-api-access-sn8qh\") pod \"127fafd2-46e8-486a-b9df-3c48b59ec61b\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.954824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stcsh\" (UniqueName: \"kubernetes.io/projected/51710375-227c-4313-b21e-ee581440728c-kube-api-access-stcsh\") pod \"51710375-227c-4313-b21e-ee581440728c\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.954846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-bundle\") pod \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.954872    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvwls\" (UniqueName: \"kubernetes.io/projected/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-kube-api-access-zvwls\") pod \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.954958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-bundle\") pod \"51710375-227c-4313-b21e-ee581440728c\" (UID: \"51710375-227c-4313-b21e-ee581440728c\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.954981    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-util\") pod \"084da37e-406c-4fd6-bb12-1479219bcf7e\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.955010    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-bundle\") pod \"127fafd2-46e8-486a-b9df-3c48b59ec61b\" (UID: \"127fafd2-46e8-486a-b9df-3c48b59ec61b\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.955216    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-util\") pod \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\" (UID: \"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.955931    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7brd\" (UniqueName: \"kubernetes.io/projected/084da37e-406c-4fd6-bb12-1479219bcf7e-kube-api-access-m7brd\") pod \"084da37e-406c-4fd6-bb12-1479219bcf7e\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.955970    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-bundle\") pod \"084da37e-406c-4fd6-bb12-1479219bcf7e\" (UID: \"084da37e-406c-4fd6-bb12-1479219bcf7e\") "
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.956041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-bundle" (OuterVolumeSpecName: "bundle") pod "51710375-227c-4313-b21e-ee581440728c" (UID: "51710375-227c-4313-b21e-ee581440728c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.956262    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.956462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-bundle" (OuterVolumeSpecName: "bundle") pod "aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" (UID: "aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:04.958408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.956685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-bundle" (OuterVolumeSpecName: "bundle") pod "127fafd2-46e8-486a-b9df-3c48b59ec61b" (UID: "127fafd2-46e8-486a-b9df-3c48b59ec61b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:04.959214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.958965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-bundle" (OuterVolumeSpecName: "bundle") pod "084da37e-406c-4fd6-bb12-1479219bcf7e" (UID: "084da37e-406c-4fd6-bb12-1479219bcf7e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:04.959638 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.959599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51710375-227c-4313-b21e-ee581440728c-kube-api-access-stcsh" (OuterVolumeSpecName: "kube-api-access-stcsh") pod "51710375-227c-4313-b21e-ee581440728c" (UID: "51710375-227c-4313-b21e-ee581440728c"). InnerVolumeSpecName "kube-api-access-stcsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:04.959984 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.959956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084da37e-406c-4fd6-bb12-1479219bcf7e-kube-api-access-m7brd" (OuterVolumeSpecName: "kube-api-access-m7brd") pod "084da37e-406c-4fd6-bb12-1479219bcf7e" (UID: "084da37e-406c-4fd6-bb12-1479219bcf7e"). InnerVolumeSpecName "kube-api-access-m7brd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:04.962010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.961972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127fafd2-46e8-486a-b9df-3c48b59ec61b-kube-api-access-sn8qh" (OuterVolumeSpecName: "kube-api-access-sn8qh") pod "127fafd2-46e8-486a-b9df-3c48b59ec61b" (UID: "127fafd2-46e8-486a-b9df-3c48b59ec61b"). InnerVolumeSpecName "kube-api-access-sn8qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:04.963120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.963094    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-kube-api-access-zvwls" (OuterVolumeSpecName: "kube-api-access-zvwls") pod "aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" (UID: "aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f"). InnerVolumeSpecName "kube-api-access-zvwls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:04.964705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.964671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-util" (OuterVolumeSpecName: "util") pod "51710375-227c-4313-b21e-ee581440728c" (UID: "51710375-227c-4313-b21e-ee581440728c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:04.965350 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.965327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-util" (OuterVolumeSpecName: "util") pod "127fafd2-46e8-486a-b9df-3c48b59ec61b" (UID: "127fafd2-46e8-486a-b9df-3c48b59ec61b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:04.965797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.965770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-util" (OuterVolumeSpecName: "util") pod "084da37e-406c-4fd6-bb12-1479219bcf7e" (UID: "084da37e-406c-4fd6-bb12-1479219bcf7e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:04.966479 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:04.966457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-util" (OuterVolumeSpecName: "util") pod "aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" (UID: "aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:05.057218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057131    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-util\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057162    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057189    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-util\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057201    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7brd\" (UniqueName: \"kubernetes.io/projected/084da37e-406c-4fd6-bb12-1479219bcf7e-kube-api-access-m7brd\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057217    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/084da37e-406c-4fd6-bb12-1479219bcf7e-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057228    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/127fafd2-46e8-486a-b9df-3c48b59ec61b-util\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057238    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51710375-227c-4313-b21e-ee581440728c-util\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057249    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn8qh\" (UniqueName: \"kubernetes.io/projected/127fafd2-46e8-486a-b9df-3c48b59ec61b-kube-api-access-sn8qh\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057261    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stcsh\" (UniqueName: \"kubernetes.io/projected/51710375-227c-4313-b21e-ee581440728c-kube-api-access-stcsh\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057272    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.057469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.057285    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwls\" (UniqueName: \"kubernetes.io/projected/aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f-kube-api-access-zvwls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:05.734092 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.734062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5" event={"ID":"084da37e-406c-4fd6-bb12-1479219bcf7e","Type":"ContainerDied","Data":"3384f59ff8c0b4275b4c30dd803c2dee090e9cfce795e379e9797b1204fb3d53"}
Sep 10 23:08:05.734264 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.734110    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3384f59ff8c0b4275b4c30dd803c2dee090e9cfce795e379e9797b1204fb3d53"
Sep 10 23:08:05.734264 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.734069    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar28h5"
Sep 10 23:08:05.736057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.736037    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh"
Sep 10 23:08:05.736187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.736069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv4xh" event={"ID":"127fafd2-46e8-486a-b9df-3c48b59ec61b","Type":"ContainerDied","Data":"a79477d80489a7a3b1f41fdd68d7208b6eca3e229bc853fcc00f0b4b1ccab847"}
Sep 10 23:08:05.736187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.736115    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a79477d80489a7a3b1f41fdd68d7208b6eca3e229bc853fcc00f0b4b1ccab847"
Sep 10 23:08:05.738000 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.737981    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5"
Sep 10 23:08:05.738125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.737999    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqqlz5" event={"ID":"51710375-227c-4313-b21e-ee581440728c","Type":"ContainerDied","Data":"34bad390488ef329a5b0365e326b16b54e77b783722dd000a9ff54138e5cec89"}
Sep 10 23:08:05.738125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.738035    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34bad390488ef329a5b0365e326b16b54e77b783722dd000a9ff54138e5cec89"
Sep 10 23:08:05.739983 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.739959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw" event={"ID":"aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f","Type":"ContainerDied","Data":"52826f9da5aee5bc7b720fe3d980affc5e1a1b207a46ab0bd697402af66355fb"}
Sep 10 23:08:05.740105 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.740002    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52826f9da5aee5bc7b720fe3d980affc5e1a1b207a46ab0bd697402af66355fb"
Sep 10 23:08:05.740105 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:05.739969    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9xvw"
Sep 10 23:08:08.941435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:08.941409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:09.942717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:09.942686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:13.179896 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:13.179859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:14.033877 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.033841    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-df766899b-4pdnr"]
Sep 10 23:08:14.034188 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034175    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerName="util"
Sep 10 23:08:14.034231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034190    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerName="util"
Sep 10 23:08:14.034231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034202    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51710375-227c-4313-b21e-ee581440728c" containerName="extract"
Sep 10 23:08:14.034231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034212    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51710375-227c-4313-b21e-ee581440728c" containerName="extract"
Sep 10 23:08:14.034231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034220    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51710375-227c-4313-b21e-ee581440728c" containerName="pull"
Sep 10 23:08:14.034231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034226    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51710375-227c-4313-b21e-ee581440728c" containerName="pull"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034239    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerName="extract"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034244    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerName="extract"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034251    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51710375-227c-4313-b21e-ee581440728c" containerName="util"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034256    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51710375-227c-4313-b21e-ee581440728c" containerName="util"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034261    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerName="extract"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034266    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerName="extract"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034276    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerName="util"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034285    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerName="util"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034294    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerName="util"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034300    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerName="util"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034307    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerName="pull"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034312    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerName="pull"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034318    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerName="pull"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034322    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerName="pull"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034328    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerName="pull"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034333    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerName="pull"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034337    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerName="extract"
Sep 10 23:08:14.034381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034342    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerName="extract"
Sep 10 23:08:14.034917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034397    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="127fafd2-46e8-486a-b9df-3c48b59ec61b" containerName="extract"
Sep 10 23:08:14.034917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034405    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51710375-227c-4313-b21e-ee581440728c" containerName="extract"
Sep 10 23:08:14.034917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034411    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="084da37e-406c-4fd6-bb12-1479219bcf7e" containerName="extract"
Sep 10 23:08:14.034917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.034416    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaefc35d-0580-4f4a-a2e4-e0e9c090fa2f" containerName="extract"
Sep 10 23:08:14.038463 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.038446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.051979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.051955    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df766899b-4pdnr"]
Sep 10 23:08:14.086346 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.086316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:14.138717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.138686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-console-serving-cert\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.138867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.138722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-console-oauth-config\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.138867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.138749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-service-ca\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.138867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.138824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-trusted-ca-bundle\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.138867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.138850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-oauth-serving-cert\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.138998 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.138881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-console-config\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.138998 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.138945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tthz6\" (UniqueName: \"kubernetes.io/projected/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-kube-api-access-tthz6\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.186573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.186512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:14.240298 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.240273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tthz6\" (UniqueName: \"kubernetes.io/projected/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-kube-api-access-tthz6\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.240427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.240326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-console-serving-cert\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.240427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.240354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-console-oauth-config\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.240427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.240375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-service-ca\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.240427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.240402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-trusted-ca-bundle\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.240640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.240597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-oauth-serving-cert\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.240686 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.240667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-console-config\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.241231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.241194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-oauth-serving-cert\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.241231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.241206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-service-ca\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.241416 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.241272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-console-config\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.241566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.241523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-trusted-ca-bundle\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.242952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.242923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-console-oauth-config\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.243130 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.243112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-console-serving-cert\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.248128 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.248106    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tthz6\" (UniqueName: \"kubernetes.io/projected/1f19623b-f7b4-4b23-a3e0-39c5adff27f7-kube-api-access-tthz6\") pod \"console-df766899b-4pdnr\" (UID: \"1f19623b-f7b4-4b23-a3e0-39c5adff27f7\") " pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.347842 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.347810    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:14.353271 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.353245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:14.630501 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.630466    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df766899b-4pdnr"]
Sep 10 23:08:14.632650 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:08:14.632616    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f19623b_f7b4_4b23_a3e0_39c5adff27f7.slice/crio-2c869b41eb6c50d8423558220f7a67a3ba45deb50aac9861fb53a735f8867f3f WatchSource:0}: Error finding container 2c869b41eb6c50d8423558220f7a67a3ba45deb50aac9861fb53a735f8867f3f: Status 404 returned error can't find the container with id 2c869b41eb6c50d8423558220f7a67a3ba45deb50aac9861fb53a735f8867f3f
Sep 10 23:08:14.787636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.787603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df766899b-4pdnr" event={"ID":"1f19623b-f7b4-4b23-a3e0-39c5adff27f7","Type":"ContainerStarted","Data":"ca761c95ba004e3dedc2683533dd8790da21257b65747f5ad7ee55443675f2a4"}
Sep 10 23:08:14.787816 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.787650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df766899b-4pdnr" event={"ID":"1f19623b-f7b4-4b23-a3e0-39c5adff27f7","Type":"ContainerStarted","Data":"2c869b41eb6c50d8423558220f7a67a3ba45deb50aac9861fb53a735f8867f3f"}
Sep 10 23:08:14.812024 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:14.811968    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-df766899b-4pdnr" podStartSLOduration=0.811950551 podStartE2EDuration="811.950551ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:14.810926964 +0000 UTC m=+394.512559909" watchObservedRunningTime="2026-09-10 23:08:14.811950551 +0000 UTC m=+394.513583497"
Sep 10 23:08:15.172697 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:15.172669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:15.274751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:15.274723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:16.320675 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.320635    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"]
Sep 10 23:08:16.324382 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.324359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:16.326944 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.326923    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 23:08:16.327516 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.327498    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-whd78\""
Sep 10 23:08:16.327617 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.327530    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 23:08:16.335152 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.335126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"]
Sep 10 23:08:16.358258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.358230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662q9\" (UniqueName: \"kubernetes.io/projected/7310c9a5-bae3-4eec-962c-fcbd98ae3b14-kube-api-access-662q9\") pod \"limitador-operator-controller-manager-74495c69f8-l4ctk\" (UID: \"7310c9a5-bae3-4eec-962c-fcbd98ae3b14\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:16.459452 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.459416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662q9\" (UniqueName: \"kubernetes.io/projected/7310c9a5-bae3-4eec-962c-fcbd98ae3b14-kube-api-access-662q9\") pod \"limitador-operator-controller-manager-74495c69f8-l4ctk\" (UID: \"7310c9a5-bae3-4eec-962c-fcbd98ae3b14\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:16.470121 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.470098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-662q9\" (UniqueName: \"kubernetes.io/projected/7310c9a5-bae3-4eec-962c-fcbd98ae3b14-kube-api-access-662q9\") pod \"limitador-operator-controller-manager-74495c69f8-l4ctk\" (UID: \"7310c9a5-bae3-4eec-962c-fcbd98ae3b14\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:16.636759 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.636735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:16.916497 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.916471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"]
Sep 10 23:08:16.918564 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:08:16.918521    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7310c9a5_bae3_4eec_962c_fcbd98ae3b14.slice/crio-bf2aa42b3e0253dadcb43fb647c04c51ba6d756caaf7ced7c3be8cce0318a4ca WatchSource:0}: Error finding container bf2aa42b3e0253dadcb43fb647c04c51ba6d756caaf7ced7c3be8cce0318a4ca: Status 404 returned error can't find the container with id bf2aa42b3e0253dadcb43fb647c04c51ba6d756caaf7ced7c3be8cce0318a4ca
Sep 10 23:08:16.952293 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:16.952265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:17.545607 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:17.545580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:17.810850 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:17.810769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk" event={"ID":"7310c9a5-bae3-4eec-962c-fcbd98ae3b14","Type":"ContainerStarted","Data":"bf2aa42b3e0253dadcb43fb647c04c51ba6d756caaf7ced7c3be8cce0318a4ca"}
Sep 10 23:08:18.158018 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:18.157974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:19.824577 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:19.824459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk" event={"ID":"7310c9a5-bae3-4eec-962c-fcbd98ae3b14","Type":"ContainerStarted","Data":"75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84"}
Sep 10 23:08:19.824577 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:19.824569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:19.853124 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:19.853066    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk" podStartSLOduration=1.207439502 podStartE2EDuration="3.853048183s" totalImagesPullingTime="2.645608681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:19.851069305 +0000 UTC m=+399.552702248" watchObservedRunningTime="2026-09-10 23:08:19.853048183 +0000 UTC m=+399.554681126"
Sep 10 23:08:23.182259 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:23.182223    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:24.348507 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:24.348465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:24.348945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:24.348549    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:24.353236 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:24.353214    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:24.853309 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:24.853282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-df766899b-4pdnr"
Sep 10 23:08:24.918478 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:24.918437    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c484964cb-7n8kl"]
Sep 10 23:08:25.085751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:25.085706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:25.184527 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:25.184436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:25.296207 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:25.296176    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:29.725308 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:29.725268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:30.079780 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:30.079679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:30.834863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:30.834833    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:34.674594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:34.674559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:34.761706 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:34.761671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:35.384427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:35.384397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:36.383639 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:36.383593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:37.444365 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:37.444320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:37.543327 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:37.543288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:40.927327 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:40.927299    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"]
Sep 10 23:08:40.940716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:40.940686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:40.944946 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:40.944921    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-thk95\""
Sep 10 23:08:40.945585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:40.945567    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"]
Sep 10 23:08:40.948109 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:40.948090    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"]
Sep 10 23:08:40.948327 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:40.948304    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk" podUID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" containerName="manager" containerID="cri-o://75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84" gracePeriod=2
Sep 10 23:08:40.969243 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:40.969218    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"]
Sep 10 23:08:40.979808 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:08:40.979771    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l4ctk\" is forbidden: User \"system:node:ip-10-0-135-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-96.ec2.internal' and this object" podUID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:41.071444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.071419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tw9d\" (UniqueName: \"kubernetes.io/projected/634d2e53-72b4-4397-9edc-8881532b2797-kube-api-access-6tw9d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vb4w2\" (UID: \"634d2e53-72b4-4397-9edc-8881532b2797\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:41.071628 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.071467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/634d2e53-72b4-4397-9edc-8881532b2797-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vb4w2\" (UID: \"634d2e53-72b4-4397-9edc-8881532b2797\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:41.169376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.169346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:41.172614 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.172591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tw9d\" (UniqueName: \"kubernetes.io/projected/634d2e53-72b4-4397-9edc-8881532b2797-kube-api-access-6tw9d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vb4w2\" (UID: \"634d2e53-72b4-4397-9edc-8881532b2797\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:41.172723 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.172659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/634d2e53-72b4-4397-9edc-8881532b2797-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vb4w2\" (UID: \"634d2e53-72b4-4397-9edc-8881532b2797\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:41.173013 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.172994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/634d2e53-72b4-4397-9edc-8881532b2797-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vb4w2\" (UID: \"634d2e53-72b4-4397-9edc-8881532b2797\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:41.180785 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.180735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tw9d\" (UniqueName: \"kubernetes.io/projected/634d2e53-72b4-4397-9edc-8881532b2797-kube-api-access-6tw9d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vb4w2\" (UID: \"634d2e53-72b4-4397-9edc-8881532b2797\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:41.184211 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.184193    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:41.188725 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:08:41.188705    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l4ctk\" is forbidden: User \"system:node:ip-10-0-135-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-96.ec2.internal' and this object" podUID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:41.254171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.254140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:41.273221 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.273195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-662q9\" (UniqueName: \"kubernetes.io/projected/7310c9a5-bae3-4eec-962c-fcbd98ae3b14-kube-api-access-662q9\") pod \"7310c9a5-bae3-4eec-962c-fcbd98ae3b14\" (UID: \"7310c9a5-bae3-4eec-962c-fcbd98ae3b14\") "
Sep 10 23:08:41.275428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.275399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7310c9a5-bae3-4eec-962c-fcbd98ae3b14-kube-api-access-662q9" (OuterVolumeSpecName: "kube-api-access-662q9") pod "7310c9a5-bae3-4eec-962c-fcbd98ae3b14" (UID: "7310c9a5-bae3-4eec-962c-fcbd98ae3b14"). InnerVolumeSpecName "kube-api-access-662q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:41.374168 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.374137    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-662q9\" (UniqueName: \"kubernetes.io/projected/7310c9a5-bae3-4eec-962c-fcbd98ae3b14-kube-api-access-662q9\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:41.536218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.536193    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"]
Sep 10 23:08:41.538387 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:08:41.538361    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634d2e53_72b4_4397_9edc_8881532b2797.slice/crio-2ed659a032bc288e31039794043b28927e188d53496dae88c33084500129ca5b WatchSource:0}: Error finding container 2ed659a032bc288e31039794043b28927e188d53496dae88c33084500129ca5b: Status 404 returned error can't find the container with id 2ed659a032bc288e31039794043b28927e188d53496dae88c33084500129ca5b
Sep 10 23:08:41.938290 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.938255    2580 generic.go:356] "Generic (PLEG): container finished" podID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" containerID="75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84" exitCode=0
Sep 10 23:08:41.938741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.938302    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:41.938741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.938373    2580 scope.go:117] "RemoveContainer" containerID="75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84"
Sep 10 23:08:41.940053 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.940028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2" event={"ID":"634d2e53-72b4-4397-9edc-8881532b2797","Type":"ContainerStarted","Data":"2ed659a032bc288e31039794043b28927e188d53496dae88c33084500129ca5b"}
Sep 10 23:08:41.940723 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:08:41.940698    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l4ctk\" is forbidden: User \"system:node:ip-10-0-135-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-96.ec2.internal' and this object" podUID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:41.948340 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.948325    2580 scope.go:117] "RemoveContainer" containerID="75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84"
Sep 10 23:08:41.948583 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:08:41.948567    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84\": container with ID starting with 75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84 not found: ID does not exist" containerID="75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84"
Sep 10 23:08:41.948654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:41.948590    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84"} err="failed to get container status \"75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84\": rpc error: code = NotFound desc = could not find container \"75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84\": container with ID starting with 75552dfd8cfcfcef5e79f8911429936af3f8295c037337ac92eab192e535ba84 not found: ID does not exist"
Sep 10 23:08:41.949886 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:08:41.949864    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l4ctk\" is forbidden: User \"system:node:ip-10-0-135-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-96.ec2.internal' and this object" podUID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l4ctk"
Sep 10 23:08:42.834421 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:42.834386    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" path="/var/lib/kubelet/pods/7310c9a5-bae3-4eec-962c-fcbd98ae3b14/volumes"
Sep 10 23:08:44.575138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:44.575111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:44.681171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:44.681135    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:45.965387 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:45.965349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2" event={"ID":"634d2e53-72b4-4397-9edc-8881532b2797","Type":"ContainerStarted","Data":"b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b"}
Sep 10 23:08:45.965806 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:45.965417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:45.985796 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:45.985717    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2" podStartSLOduration=2.547885872 podStartE2EDuration="5.985698152s" totalImagesPullingTime="3.43781228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:45.984862615 +0000 UTC m=+425.686495567" watchObservedRunningTime="2026-09-10 23:08:45.985698152 +0000 UTC m=+425.687331098"
Sep 10 23:08:49.179637 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:49.179609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:49.247715 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:49.247687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:49.372852 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:49.372820    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:49.433002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:49.432919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:49.938399 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:49.938360    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c484964cb-7n8kl" podUID="446d705d-2b22-4de2-9576-bcea82af0097" containerName="console" containerID="cri-o://8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a" gracePeriod=15
Sep 10 23:08:50.199228 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.199171    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:08:50.252471 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.252445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-service-ca\") pod \"446d705d-2b22-4de2-9576-bcea82af0097\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") "
Sep 10 23:08:50.252654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.252477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-serving-cert\") pod \"446d705d-2b22-4de2-9576-bcea82af0097\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") "
Sep 10 23:08:50.252654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.252495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-console-config\") pod \"446d705d-2b22-4de2-9576-bcea82af0097\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") "
Sep 10 23:08:50.252654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.252519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-oauth-serving-cert\") pod \"446d705d-2b22-4de2-9576-bcea82af0097\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") "
Sep 10 23:08:50.252654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.252568    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-trusted-ca-bundle\") pod \"446d705d-2b22-4de2-9576-bcea82af0097\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") "
Sep 10 23:08:50.252654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.252600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr5pt\" (UniqueName: \"kubernetes.io/projected/446d705d-2b22-4de2-9576-bcea82af0097-kube-api-access-xr5pt\") pod \"446d705d-2b22-4de2-9576-bcea82af0097\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") "
Sep 10 23:08:50.252654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.252624    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-oauth-config\") pod \"446d705d-2b22-4de2-9576-bcea82af0097\" (UID: \"446d705d-2b22-4de2-9576-bcea82af0097\") "
Sep 10 23:08:50.253007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.252984    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-console-config" (OuterVolumeSpecName: "console-config") pod "446d705d-2b22-4de2-9576-bcea82af0097" (UID: "446d705d-2b22-4de2-9576-bcea82af0097"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:08:50.253067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.253000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-service-ca" (OuterVolumeSpecName: "service-ca") pod "446d705d-2b22-4de2-9576-bcea82af0097" (UID: "446d705d-2b22-4de2-9576-bcea82af0097"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:08:50.253275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.253247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "446d705d-2b22-4de2-9576-bcea82af0097" (UID: "446d705d-2b22-4de2-9576-bcea82af0097"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:08:50.253654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.253632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "446d705d-2b22-4de2-9576-bcea82af0097" (UID: "446d705d-2b22-4de2-9576-bcea82af0097"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:08:50.254987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.254967    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "446d705d-2b22-4de2-9576-bcea82af0097" (UID: "446d705d-2b22-4de2-9576-bcea82af0097"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:08:50.255144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.255088    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446d705d-2b22-4de2-9576-bcea82af0097-kube-api-access-xr5pt" (OuterVolumeSpecName: "kube-api-access-xr5pt") pod "446d705d-2b22-4de2-9576-bcea82af0097" (UID: "446d705d-2b22-4de2-9576-bcea82af0097"). InnerVolumeSpecName "kube-api-access-xr5pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:50.255144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.255093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "446d705d-2b22-4de2-9576-bcea82af0097" (UID: "446d705d-2b22-4de2-9576-bcea82af0097"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:08:50.353241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.353213    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:50.353241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.353239    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-serving-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:50.353241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.353248    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-console-config\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:50.353447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.353258    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-oauth-serving-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:50.353447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.353265    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/446d705d-2b22-4de2-9576-bcea82af0097-trusted-ca-bundle\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:50.353447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.353274    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr5pt\" (UniqueName: \"kubernetes.io/projected/446d705d-2b22-4de2-9576-bcea82af0097-kube-api-access-xr5pt\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:50.353447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.353282    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/446d705d-2b22-4de2-9576-bcea82af0097-console-oauth-config\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:50.990629 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.990591    2580 generic.go:356] "Generic (PLEG): container finished" podID="446d705d-2b22-4de2-9576-bcea82af0097" containerID="8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a" exitCode=0
Sep 10 23:08:50.990818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.990655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c484964cb-7n8kl"
Sep 10 23:08:50.990818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.990679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c484964cb-7n8kl" event={"ID":"446d705d-2b22-4de2-9576-bcea82af0097","Type":"ContainerDied","Data":"8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a"}
Sep 10 23:08:50.990818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.990752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c484964cb-7n8kl" event={"ID":"446d705d-2b22-4de2-9576-bcea82af0097","Type":"ContainerDied","Data":"710a4ccb5396e750c706ac221f9b812be91a0268240f5c1e29eb71c666428389"}
Sep 10 23:08:50.990818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.990772    2580 scope.go:117] "RemoveContainer" containerID="8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a"
Sep 10 23:08:50.999483 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.999464    2580 scope.go:117] "RemoveContainer" containerID="8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a"
Sep 10 23:08:50.999776 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:08:50.999757    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a\": container with ID starting with 8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a not found: ID does not exist" containerID="8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a"
Sep 10 23:08:50.999835 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:50.999784    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a"} err="failed to get container status \"8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a\": rpc error: code = NotFound desc = could not find container \"8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a\": container with ID starting with 8677c305d41ef36cddb6ce7286d8a9cba58b716af0f4c9c634ddcdfc94e8568a not found: ID does not exist"
Sep 10 23:08:51.008438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:51.008411    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c484964cb-7n8kl"]
Sep 10 23:08:51.013493 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:51.013472    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c484964cb-7n8kl"]
Sep 10 23:08:51.310519 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:51.310484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:51.424486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:51.424452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:52.834334 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:52.834299    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="446d705d-2b22-4de2-9576-bcea82af0097" path="/var/lib/kubelet/pods/446d705d-2b22-4de2-9576-bcea82af0097/volumes"
Sep 10 23:08:56.053573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:56.053533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:56.153386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:56.153354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:56.256506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:56.256469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:56.972978 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:56.972945    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:57.102742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.102712    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"]
Sep 10 23:08:57.103201 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.102987    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2" podUID="634d2e53-72b4-4397-9edc-8881532b2797" containerName="manager" containerID="cri-o://b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b" gracePeriod=10
Sep 10 23:08:57.347052 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.347028    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:57.367812 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.367787    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"]
Sep 10 23:08:57.368120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.368099    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446d705d-2b22-4de2-9576-bcea82af0097" containerName="console"
Sep 10 23:08:57.368120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.368115    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="446d705d-2b22-4de2-9576-bcea82af0097" containerName="console"
Sep 10 23:08:57.368252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.368138    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634d2e53-72b4-4397-9edc-8881532b2797" containerName="manager"
Sep 10 23:08:57.368252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.368147    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="634d2e53-72b4-4397-9edc-8881532b2797" containerName="manager"
Sep 10 23:08:57.368252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.368159    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" containerName="manager"
Sep 10 23:08:57.368252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.368164    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" containerName="manager"
Sep 10 23:08:57.368252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.368220    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="446d705d-2b22-4de2-9576-bcea82af0097" containerName="console"
Sep 10 23:08:57.368252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.368234    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7310c9a5-bae3-4eec-962c-fcbd98ae3b14" containerName="manager"
Sep 10 23:08:57.368252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.368240    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="634d2e53-72b4-4397-9edc-8881532b2797" containerName="manager"
Sep 10 23:08:57.371196 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.371181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:08:57.388096 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.388072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"]
Sep 10 23:08:57.398961 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.398932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgczn\" (UniqueName: \"kubernetes.io/projected/2a96ae29-51f6-411b-adc0-f9f737923d12-kube-api-access-pgczn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dz55b\" (UID: \"2a96ae29-51f6-411b-adc0-f9f737923d12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:08:57.399073 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.399030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a96ae29-51f6-411b-adc0-f9f737923d12-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dz55b\" (UID: \"2a96ae29-51f6-411b-adc0-f9f737923d12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:08:57.500081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.500050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tw9d\" (UniqueName: \"kubernetes.io/projected/634d2e53-72b4-4397-9edc-8881532b2797-kube-api-access-6tw9d\") pod \"634d2e53-72b4-4397-9edc-8881532b2797\" (UID: \"634d2e53-72b4-4397-9edc-8881532b2797\") "
Sep 10 23:08:57.500081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.500082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/634d2e53-72b4-4397-9edc-8881532b2797-extensions-socket-volume\") pod \"634d2e53-72b4-4397-9edc-8881532b2797\" (UID: \"634d2e53-72b4-4397-9edc-8881532b2797\") "
Sep 10 23:08:57.500280 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.500212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgczn\" (UniqueName: \"kubernetes.io/projected/2a96ae29-51f6-411b-adc0-f9f737923d12-kube-api-access-pgczn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dz55b\" (UID: \"2a96ae29-51f6-411b-adc0-f9f737923d12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:08:57.500280 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.500259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a96ae29-51f6-411b-adc0-f9f737923d12-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dz55b\" (UID: \"2a96ae29-51f6-411b-adc0-f9f737923d12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:08:57.500674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.500643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a96ae29-51f6-411b-adc0-f9f737923d12-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dz55b\" (UID: \"2a96ae29-51f6-411b-adc0-f9f737923d12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:08:57.500787 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.500715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634d2e53-72b4-4397-9edc-8881532b2797-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "634d2e53-72b4-4397-9edc-8881532b2797" (UID: "634d2e53-72b4-4397-9edc-8881532b2797"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:57.502258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.502237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634d2e53-72b4-4397-9edc-8881532b2797-kube-api-access-6tw9d" (OuterVolumeSpecName: "kube-api-access-6tw9d") pod "634d2e53-72b4-4397-9edc-8881532b2797" (UID: "634d2e53-72b4-4397-9edc-8881532b2797"). InnerVolumeSpecName "kube-api-access-6tw9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:57.510674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.510651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgczn\" (UniqueName: \"kubernetes.io/projected/2a96ae29-51f6-411b-adc0-f9f737923d12-kube-api-access-pgczn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dz55b\" (UID: \"2a96ae29-51f6-411b-adc0-f9f737923d12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:08:57.601576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.601534    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tw9d\" (UniqueName: \"kubernetes.io/projected/634d2e53-72b4-4397-9edc-8881532b2797-kube-api-access-6tw9d\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:57.601576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.601573    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/634d2e53-72b4-4397-9edc-8881532b2797-extensions-socket-volume\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:08:57.680882 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.680858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:08:57.961432 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:57.961406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"]
Sep 10 23:08:57.963603 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:08:57.963559    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a96ae29_51f6_411b_adc0_f9f737923d12.slice/crio-9ef993ecd41aefb5f7a2cae575c982f8dbccc985d76b8f7d1a6c4a6b96cb49b3 WatchSource:0}: Error finding container 9ef993ecd41aefb5f7a2cae575c982f8dbccc985d76b8f7d1a6c4a6b96cb49b3: Status 404 returned error can't find the container with id 9ef993ecd41aefb5f7a2cae575c982f8dbccc985d76b8f7d1a6c4a6b96cb49b3
Sep 10 23:08:58.030420 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.030389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b" event={"ID":"2a96ae29-51f6-411b-adc0-f9f737923d12","Type":"ContainerStarted","Data":"f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd"}
Sep 10 23:08:58.030601 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.030449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b" event={"ID":"2a96ae29-51f6-411b-adc0-f9f737923d12","Type":"ContainerStarted","Data":"9ef993ecd41aefb5f7a2cae575c982f8dbccc985d76b8f7d1a6c4a6b96cb49b3"}
Sep 10 23:08:58.030601 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.030507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:08:58.031967 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.031944    2580 generic.go:356] "Generic (PLEG): container finished" podID="634d2e53-72b4-4397-9edc-8881532b2797" containerID="b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b" exitCode=0
Sep 10 23:08:58.032076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.031975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2" event={"ID":"634d2e53-72b4-4397-9edc-8881532b2797","Type":"ContainerDied","Data":"b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b"}
Sep 10 23:08:58.032076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.031989    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"
Sep 10 23:08:58.032076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.032004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2" event={"ID":"634d2e53-72b4-4397-9edc-8881532b2797","Type":"ContainerDied","Data":"2ed659a032bc288e31039794043b28927e188d53496dae88c33084500129ca5b"}
Sep 10 23:08:58.032076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.032025    2580 scope.go:117] "RemoveContainer" containerID="b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b"
Sep 10 23:08:58.041102 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.041083    2580 scope.go:117] "RemoveContainer" containerID="b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b"
Sep 10 23:08:58.041352 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:08:58.041337    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b\": container with ID starting with b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b not found: ID does not exist" containerID="b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b"
Sep 10 23:08:58.041408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.041359    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b"} err="failed to get container status \"b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b\": rpc error: code = NotFound desc = could not find container \"b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b\": container with ID starting with b8182c097dfa44a70837f3a4979978e3946980f4e2629361ef8aa531348a514b not found: ID does not exist"
Sep 10 23:08:58.053333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.053285    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b" podStartSLOduration=1.053270413 podStartE2EDuration="1.053270413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:58.05070922 +0000 UTC m=+437.752342164" watchObservedRunningTime="2026-09-10 23:08:58.053270413 +0000 UTC m=+437.754903353"
Sep 10 23:08:58.072266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.072240    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"]
Sep 10 23:08:58.075204 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.075181    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vb4w2"]
Sep 10 23:08:58.833508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:08:58.833471    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="634d2e53-72b4-4397-9edc-8881532b2797" path="/var/lib/kubelet/pods/634d2e53-72b4-4397-9edc-8881532b2797/volumes"
Sep 10 23:09:02.533280 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:02.533200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:02.628762 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:02.628728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:08.083451 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:08.083417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:08.139351 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:08.139314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:08.682348 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:08.682318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:08.783695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:08.783668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:09.041338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:09.041262    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:09:12.087501 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:12.087467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:12.188728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:12.188694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:13.464444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.464404    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"]
Sep 10 23:09:13.468241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.468221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.470706 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.470677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rkpd4\""
Sep 10 23:09:13.487087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.487060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"]
Sep 10 23:09:13.489082 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.489058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:13.535957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.535925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.535957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.535962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418cef94-9a26-4377-b280-7edba1711bdb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.536214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.535992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418cef94-9a26-4377-b280-7edba1711bdb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.536214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.536118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.536214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.536146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.536214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.536166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418cef94-9a26-4377-b280-7edba1711bdb-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.536214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.536187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.536443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.536260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9cw5\" (UniqueName: \"kubernetes.io/projected/418cef94-9a26-4377-b280-7edba1711bdb-kube-api-access-m9cw5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.536443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.536308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.592476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.592444    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:13.637754 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.637714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.637900 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.637791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418cef94-9a26-4377-b280-7edba1711bdb-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.637900 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.637825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.637900 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.637854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9cw5\" (UniqueName: \"kubernetes.io/projected/418cef94-9a26-4377-b280-7edba1711bdb-kube-api-access-m9cw5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.637900 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.637891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.638119 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.637938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.638119 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.637947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.638119 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.637977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418cef94-9a26-4377-b280-7edba1711bdb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.638119 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.638027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418cef94-9a26-4377-b280-7edba1711bdb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.638355 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.638120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.638355 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.638143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.638355 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.638159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.638589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.638568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.638753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.638725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418cef94-9a26-4377-b280-7edba1711bdb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.640641 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.640619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418cef94-9a26-4377-b280-7edba1711bdb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.641127 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.641111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418cef94-9a26-4377-b280-7edba1711bdb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.650944 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.650920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418cef94-9a26-4377-b280-7edba1711bdb-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.653041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.653021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9cw5\" (UniqueName: \"kubernetes.io/projected/418cef94-9a26-4377-b280-7edba1711bdb-kube-api-access-m9cw5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-j687t\" (UID: \"418cef94-9a26-4377-b280-7edba1711bdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:13.779115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:13.779026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:14.061103 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:14.061079    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"]
Sep 10 23:09:14.063116 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:09:14.063091    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418cef94_9a26_4377_b280_7edba1711bdb.slice/crio-8210a5c41c12604e487b1f4f25ec1b96c8d0e87c8402daed4f57d3bbd734e329 WatchSource:0}: Error finding container 8210a5c41c12604e487b1f4f25ec1b96c8d0e87c8402daed4f57d3bbd734e329: Status 404 returned error can't find the container with id 8210a5c41c12604e487b1f4f25ec1b96c8d0e87c8402daed4f57d3bbd734e329
Sep 10 23:09:14.065104 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:14.065065    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:09:14.065171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:14.065134    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:09:14.065171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:14.065164    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:09:14.117138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:14.117114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" event={"ID":"418cef94-9a26-4377-b280-7edba1711bdb","Type":"ContainerStarted","Data":"8210a5c41c12604e487b1f4f25ec1b96c8d0e87c8402daed4f57d3bbd734e329"}
Sep 10 23:09:15.123824 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:15.123788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" event={"ID":"418cef94-9a26-4377-b280-7edba1711bdb","Type":"ContainerStarted","Data":"0fee50cd603c6a0f5de3373e8d144821787373459a629a3d5a948b642ccbd52d"}
Sep 10 23:09:15.147257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:15.147207    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" podStartSLOduration=2.147190709 podStartE2EDuration="2.147190709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:09:15.144852805 +0000 UTC m=+454.846485749" watchObservedRunningTime="2026-09-10 23:09:15.147190709 +0000 UTC m=+454.848823652"
Sep 10 23:09:15.779524 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:15.779487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:15.784733 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:15.784709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:16.129124 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:16.129093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:16.130205 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:16.130185    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:09:16.297767 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:16.297730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:16.354671 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:16.354641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:16.492751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:16.492658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:16.593644 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:16.593612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:16.696570 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:16.696526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:16.804816 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:16.804729    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:22.693753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:22.693716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:22.754877 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:22.754840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:25.560752 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:25.560717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:27.099416 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.099382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:27.225322 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.225285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:27.328011 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.327979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:27.382774 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.382691    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:09:27.385724 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.385702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:27.387787 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.387765    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 23:09:27.387890 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.387874    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6kc25\""
Sep 10 23:09:27.397587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.397567    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:09:27.425567 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.425521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:09:27.467693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.467660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17d0b358-cf06-4070-a8e1-1c2b83e9edba-config-file\") pod \"limitador-limitador-84994f7ffc-9kslx\" (UID: \"17d0b358-cf06-4070-a8e1-1c2b83e9edba\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:27.467693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.467703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxll\" (UniqueName: \"kubernetes.io/projected/17d0b358-cf06-4070-a8e1-1c2b83e9edba-kube-api-access-cpxll\") pod \"limitador-limitador-84994f7ffc-9kslx\" (UID: \"17d0b358-cf06-4070-a8e1-1c2b83e9edba\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:27.569095 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.569060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17d0b358-cf06-4070-a8e1-1c2b83e9edba-config-file\") pod \"limitador-limitador-84994f7ffc-9kslx\" (UID: \"17d0b358-cf06-4070-a8e1-1c2b83e9edba\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:27.569270 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.569103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxll\" (UniqueName: \"kubernetes.io/projected/17d0b358-cf06-4070-a8e1-1c2b83e9edba-kube-api-access-cpxll\") pod \"limitador-limitador-84994f7ffc-9kslx\" (UID: \"17d0b358-cf06-4070-a8e1-1c2b83e9edba\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:27.569749 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.569730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17d0b358-cf06-4070-a8e1-1c2b83e9edba-config-file\") pod \"limitador-limitador-84994f7ffc-9kslx\" (UID: \"17d0b358-cf06-4070-a8e1-1c2b83e9edba\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:27.586242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.586207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxll\" (UniqueName: \"kubernetes.io/projected/17d0b358-cf06-4070-a8e1-1c2b83e9edba-kube-api-access-cpxll\") pod \"limitador-limitador-84994f7ffc-9kslx\" (UID: \"17d0b358-cf06-4070-a8e1-1c2b83e9edba\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:27.697568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.697460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:27.980089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:27.980047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:09:27.980397 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:09:27.980376    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d0b358_cf06_4070_a8e1_1c2b83e9edba.slice/crio-2da05a3122cd5aa53aa340c3906dfb5fbb907d4ae325f23d6ce0b7fcf73ca513 WatchSource:0}: Error finding container 2da05a3122cd5aa53aa340c3906dfb5fbb907d4ae325f23d6ce0b7fcf73ca513: Status 404 returned error can't find the container with id 2da05a3122cd5aa53aa340c3906dfb5fbb907d4ae325f23d6ce0b7fcf73ca513
Sep 10 23:09:28.015433 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:28.015399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:28.114580 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:28.114506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:28.193286 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:28.193250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx" event={"ID":"17d0b358-cf06-4070-a8e1-1c2b83e9edba","Type":"ContainerStarted","Data":"2da05a3122cd5aa53aa340c3906dfb5fbb907d4ae325f23d6ce0b7fcf73ca513"}
Sep 10 23:09:29.200500 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:29.200451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx" event={"ID":"17d0b358-cf06-4070-a8e1-1c2b83e9edba","Type":"ContainerStarted","Data":"475e71133015e0e4a9e66e66f8210ef447d10b762e71c7109c07e47c3d97b345"}
Sep 10 23:09:29.200971 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:29.200557    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:29.221990 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:29.221939    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx" podStartSLOduration=1.196663196 podStartE2EDuration="2.221924225s" totalImagesPullingTime="1.025261029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:09:29.219750699 +0000 UTC m=+468.921383642" watchObservedRunningTime="2026-09-10 23:09:29.221924225 +0000 UTC m=+468.923557169"
Sep 10 23:09:40.207202 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:40.207174    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-9kslx"
Sep 10 23:09:42.444176 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:42.444128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:42.543652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:42.543624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:42.918618 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:42.918584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:55.676155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:55.676124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:55.816681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:55.816650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:55.908249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:55.908217    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-5ltwz"]
Sep 10 23:09:55.911773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:55.911745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5ltwz"
Sep 10 23:09:55.914812 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:55.914790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7cz9b\""
Sep 10 23:09:55.923805 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:55.923782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-5ltwz"]
Sep 10 23:09:55.990021 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:55.989937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5ltwz"]
Sep 10 23:09:55.990212 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:09:55.990193    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-k5r6b], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-5ltwz" podUID="f1c00e28-3fde-4793-ae33-bb7fd2981ea7"
Sep 10 23:09:55.999915 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:55.999880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5r6b\" (UniqueName: \"kubernetes.io/projected/f1c00e28-3fde-4793-ae33-bb7fd2981ea7-kube-api-access-k5r6b\") pod \"authorino-557f876f69-5ltwz\" (UID: \"f1c00e28-3fde-4793-ae33-bb7fd2981ea7\") " pod="kuadrant-system/authorino-557f876f69-5ltwz"
Sep 10 23:09:56.100915 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.100886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5r6b\" (UniqueName: \"kubernetes.io/projected/f1c00e28-3fde-4793-ae33-bb7fd2981ea7-kube-api-access-k5r6b\") pod \"authorino-557f876f69-5ltwz\" (UID: \"f1c00e28-3fde-4793-ae33-bb7fd2981ea7\") " pod="kuadrant-system/authorino-557f876f69-5ltwz"
Sep 10 23:09:56.110863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.110835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5r6b\" (UniqueName: \"kubernetes.io/projected/f1c00e28-3fde-4793-ae33-bb7fd2981ea7-kube-api-access-k5r6b\") pod \"authorino-557f876f69-5ltwz\" (UID: \"f1c00e28-3fde-4793-ae33-bb7fd2981ea7\") " pod="kuadrant-system/authorino-557f876f69-5ltwz"
Sep 10 23:09:56.177691 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.177649    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75cf5b7ffc-sjxll"]
Sep 10 23:09:56.181048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.181031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:09:56.183440 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.183419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 23:09:56.190642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.190617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75cf5b7ffc-sjxll"]
Sep 10 23:09:56.201763 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.201726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkrr\" (UniqueName: \"kubernetes.io/projected/27529e5a-96d1-40c6-9677-5b1776287655-kube-api-access-kfkrr\") pod \"authorino-75cf5b7ffc-sjxll\" (UID: \"27529e5a-96d1-40c6-9677-5b1776287655\") " pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:09:56.201911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.201839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27529e5a-96d1-40c6-9677-5b1776287655-tls-cert\") pod \"authorino-75cf5b7ffc-sjxll\" (UID: \"27529e5a-96d1-40c6-9677-5b1776287655\") " pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:09:56.303045 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.302956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkrr\" (UniqueName: \"kubernetes.io/projected/27529e5a-96d1-40c6-9677-5b1776287655-kube-api-access-kfkrr\") pod \"authorino-75cf5b7ffc-sjxll\" (UID: \"27529e5a-96d1-40c6-9677-5b1776287655\") " pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:09:56.303045 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.303018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27529e5a-96d1-40c6-9677-5b1776287655-tls-cert\") pod \"authorino-75cf5b7ffc-sjxll\" (UID: \"27529e5a-96d1-40c6-9677-5b1776287655\") " pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:09:56.305694 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.305671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27529e5a-96d1-40c6-9677-5b1776287655-tls-cert\") pod \"authorino-75cf5b7ffc-sjxll\" (UID: \"27529e5a-96d1-40c6-9677-5b1776287655\") " pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:09:56.312052 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.312029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkrr\" (UniqueName: \"kubernetes.io/projected/27529e5a-96d1-40c6-9677-5b1776287655-kube-api-access-kfkrr\") pod \"authorino-75cf5b7ffc-sjxll\" (UID: \"27529e5a-96d1-40c6-9677-5b1776287655\") " pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:09:56.339161 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.339135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5ltwz"
Sep 10 23:09:56.346137 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.346120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5ltwz"
Sep 10 23:09:56.404074 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.404043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5r6b\" (UniqueName: \"kubernetes.io/projected/f1c00e28-3fde-4793-ae33-bb7fd2981ea7-kube-api-access-k5r6b\") pod \"f1c00e28-3fde-4793-ae33-bb7fd2981ea7\" (UID: \"f1c00e28-3fde-4793-ae33-bb7fd2981ea7\") "
Sep 10 23:09:56.406453 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.406421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c00e28-3fde-4793-ae33-bb7fd2981ea7-kube-api-access-k5r6b" (OuterVolumeSpecName: "kube-api-access-k5r6b") pod "f1c00e28-3fde-4793-ae33-bb7fd2981ea7" (UID: "f1c00e28-3fde-4793-ae33-bb7fd2981ea7"). InnerVolumeSpecName "kube-api-access-k5r6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:09:56.491896 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.491848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:09:56.505094 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.505070    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5r6b\" (UniqueName: \"kubernetes.io/projected/f1c00e28-3fde-4793-ae33-bb7fd2981ea7-kube-api-access-k5r6b\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:09:56.762955 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:56.762932    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75cf5b7ffc-sjxll"]
Sep 10 23:09:56.764680 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:09:56.764653    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27529e5a_96d1_40c6_9677_5b1776287655.slice/crio-cccf707737166082aa044a0ed226aae451ad5219aa5a71bce082098f1ffd258c WatchSource:0}: Error finding container cccf707737166082aa044a0ed226aae451ad5219aa5a71bce082098f1ffd258c: Status 404 returned error can't find the container with id cccf707737166082aa044a0ed226aae451ad5219aa5a71bce082098f1ffd258c
Sep 10 23:09:57.345887 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:57.345859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5ltwz"
Sep 10 23:09:57.346089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:57.345858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll" event={"ID":"27529e5a-96d1-40c6-9677-5b1776287655","Type":"ContainerStarted","Data":"cccf707737166082aa044a0ed226aae451ad5219aa5a71bce082098f1ffd258c"}
Sep 10 23:09:57.382902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:57.382847    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5ltwz"]
Sep 10 23:09:57.385797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:57.385767    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-5ltwz"]
Sep 10 23:09:58.833916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:58.833885    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c00e28-3fde-4793-ae33-bb7fd2981ea7" path="/var/lib/kubelet/pods/f1c00e28-3fde-4793-ae33-bb7fd2981ea7/volumes"
Sep 10 23:09:59.358239 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:59.358209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll" event={"ID":"27529e5a-96d1-40c6-9677-5b1776287655","Type":"ContainerStarted","Data":"72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc"}
Sep 10 23:09:59.375601 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:09:59.375523    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll" podStartSLOduration=1.8245796520000002 podStartE2EDuration="3.375507967s" totalImagesPullingTime="1.550928315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:09:59.374533742 +0000 UTC m=+499.076166686" watchObservedRunningTime="2026-09-10 23:09:59.375507967 +0000 UTC m=+499.077140909"
Sep 10 23:10:04.093809 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:04.093775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:04.168617 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:04.168574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:04.224826 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:04.224794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:04.300642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:04.300607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:04.391962 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:04.391931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:06.568758 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:06.568722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:07.714098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:07.714064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:07.813822 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:07.813750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:37.701100 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:37.701025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:39.409530 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:39.409498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:39.470372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:39.470344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:39.543382 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:39.543343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:39.642337 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:39.642306    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:40.988736 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:40.988699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:41.087184 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:41.087141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:41.286828 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:41.286741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:41.385855 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:41.385828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:41.615402 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:41.615376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:41.719331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:41.719299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:41.886093 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:41.885995    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:10:51.598846 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:51.598813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:51.648227 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:51.648201    2580 generic.go:356] "Generic (PLEG): container finished" podID="418cef94-9a26-4377-b280-7edba1711bdb" containerID="0fee50cd603c6a0f5de3373e8d144821787373459a629a3d5a948b642ccbd52d" exitCode=0
Sep 10 23:10:51.648388 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:51.648271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" event={"ID":"418cef94-9a26-4377-b280-7edba1711bdb","Type":"ContainerDied","Data":"0fee50cd603c6a0f5de3373e8d144821787373459a629a3d5a948b642ccbd52d"}
Sep 10 23:10:51.648662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:51.648649    2580 scope.go:117] "RemoveContainer" containerID="0fee50cd603c6a0f5de3373e8d144821787373459a629a3d5a948b642ccbd52d"
Sep 10 23:10:51.655333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:51.650147    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:10:51.655333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:51.650316    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:10:51.655333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:51.650420    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:10:51.680566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:51.680525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:10:52.656155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:52.656117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" event={"ID":"418cef94-9a26-4377-b280-7edba1711bdb","Type":"ContainerStarted","Data":"fac6bf2717a22c73eeb9ac148f8141e04dbce67c751aee3874efb3a7032af90d"}
Sep 10 23:10:52.779844 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:52.779802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:10:52.780114 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:52.780090    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-j687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 23:10:52.780163 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:52.780144    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" podUID="418cef94-9a26-4377-b280-7edba1711bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 23:10:53.779794 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:53.779762    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-j687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 23:10:53.780162 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:53.779824    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" podUID="418cef94-9a26-4377-b280-7edba1711bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 23:10:54.780115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:54.780081    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-j687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 23:10:54.780502 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:54.780135    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" podUID="418cef94-9a26-4377-b280-7edba1711bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 23:10:55.780242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:55.780201    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-j687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 23:10:55.780689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:55.780258    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" podUID="418cef94-9a26-4377-b280-7edba1711bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 23:10:56.779742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:56.779701    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-j687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 23:10:56.779957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:56.779766    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" podUID="418cef94-9a26-4377-b280-7edba1711bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 23:10:57.779499 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:57.779466    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-j687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 23:10:57.779896 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:57.779519    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t" podUID="418cef94-9a26-4377-b280-7edba1711bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 23:10:58.784095 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:58.784063    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:10:58.784461 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:58.784322    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:10:58.784983 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:10:58.784966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-j687t"
Sep 10 23:11:17.194772 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:11:17.194730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:17.685960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:11:17.685927    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:11:22.394104 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:11:22.394071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:23.074269 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:11:23.074238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:11:34.284376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:11:34.284343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:35.082911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:11:35.082855    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:11:58.186997 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:11:58.186964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:59.385009 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:11:59.384979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:12:05.739626 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:05.739529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:10.592021 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:10.591989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:12.078385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:12.078350    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:12:22.689756 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:22.689715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:24.473758 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:24.473728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:12:34.955517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.955486    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"]
Sep 10 23:12:34.958912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.958894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:34.960997 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.960977    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 23:12:34.961865 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.961845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 23:12:34.961990 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.961847    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dnpmw\""
Sep 10 23:12:34.961990 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.961851    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 23:12:34.968416 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.968391    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"]
Sep 10 23:12:34.975272 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.975251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:34.993108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.993088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:34.993223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.993115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl94p\" (UniqueName: \"kubernetes.io/projected/39d38071-e474-463b-8e46-8761f3796170-kube-api-access-jl94p\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:34.993223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.993136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:34.993519 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.993220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:34.993519 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.993289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:34.993519 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:34.993308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d38071-e474-463b-8e46-8761f3796170-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.094099 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.094067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.094099 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.094108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl94p\" (UniqueName: \"kubernetes.io/projected/39d38071-e474-463b-8e46-8761f3796170-kube-api-access-jl94p\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.094362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.094135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.094362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.094195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.094362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.094263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.094362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.094287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d38071-e474-463b-8e46-8761f3796170-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.094618 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.094591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.094685 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.094615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.094685 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.094651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.096455 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.096436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d38071-e474-463b-8e46-8761f3796170-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.096802 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.096783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d38071-e474-463b-8e46-8761f3796170-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.101787 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.101768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl94p\" (UniqueName: \"kubernetes.io/projected/39d38071-e474-463b-8e46-8761f3796170-kube-api-access-jl94p\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-brkkz\" (UID: \"39d38071-e474-463b-8e46-8761f3796170\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.269792 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.269705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:35.546271 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.546243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"]
Sep 10 23:12:35.547617 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:12:35.547587    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39d38071_e474_463b_8e46_8761f3796170.slice/crio-c0b6e6fd9ef8dfee4d8d29210c54b18d72435b37b4157bb0947f6a3399c38b15 WatchSource:0}: Error finding container c0b6e6fd9ef8dfee4d8d29210c54b18d72435b37b4157bb0947f6a3399c38b15: Status 404 returned error can't find the container with id c0b6e6fd9ef8dfee4d8d29210c54b18d72435b37b4157bb0947f6a3399c38b15
Sep 10 23:12:35.549047 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:35.549026    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:12:36.201829 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:36.201791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz" event={"ID":"39d38071-e474-463b-8e46-8761f3796170","Type":"ContainerStarted","Data":"c0b6e6fd9ef8dfee4d8d29210c54b18d72435b37b4157bb0947f6a3399c38b15"}
Sep 10 23:12:37.209382 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:37.209340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz" event={"ID":"39d38071-e474-463b-8e46-8761f3796170","Type":"ContainerStarted","Data":"8b673ae2281b50c00f797f725d3ec71234a89d835b5d8e1bf42216d17295dac8"}
Sep 10 23:12:37.228316 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:37.228268    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz" podStartSLOduration=2.251954567 podStartE2EDuration="3.228253859s" totalImagesPullingTime="976.299292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:37.225773545 +0000 UTC m=+656.927406488" watchObservedRunningTime="2026-09-10 23:12:37.228253859 +0000 UTC m=+656.929886798"
Sep 10 23:12:37.276155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:37.276112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:12:45.270764 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:45.270734    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:45.271130 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:45.270974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:45.282944 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:45.282925    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:12:46.276306 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:12:46.276281    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-brkkz"
Sep 10 23:13:19.447159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:19.447128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:19.547877 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:19.547846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:21.517970 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:21.517937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:50.325449 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.325364    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75cf5b7ffc-sjxll"]
Sep 10 23:13:50.326045 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.325599    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll" podUID="27529e5a-96d1-40c6-9677-5b1776287655" containerName="authorino" containerID="cri-o://72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc" gracePeriod=30
Sep 10 23:13:50.575387 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.575360    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:13:50.605094 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.605063    2580 generic.go:356] "Generic (PLEG): container finished" podID="27529e5a-96d1-40c6-9677-5b1776287655" containerID="72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc" exitCode=0
Sep 10 23:13:50.605284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.605115    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll"
Sep 10 23:13:50.605284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.605160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll" event={"ID":"27529e5a-96d1-40c6-9677-5b1776287655","Type":"ContainerDied","Data":"72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc"}
Sep 10 23:13:50.605284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.605200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75cf5b7ffc-sjxll" event={"ID":"27529e5a-96d1-40c6-9677-5b1776287655","Type":"ContainerDied","Data":"cccf707737166082aa044a0ed226aae451ad5219aa5a71bce082098f1ffd258c"}
Sep 10 23:13:50.605284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.605219    2580 scope.go:117] "RemoveContainer" containerID="72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc"
Sep 10 23:13:50.614253 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.614232    2580 scope.go:117] "RemoveContainer" containerID="72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc"
Sep 10 23:13:50.614592 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:13:50.614567    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc\": container with ID starting with 72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc not found: ID does not exist" containerID="72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc"
Sep 10 23:13:50.614697 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.614603    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc"} err="failed to get container status \"72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc\": rpc error: code = NotFound desc = could not find container \"72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc\": container with ID starting with 72827fe17bd3a1c10c93d04a9f10e98c6479d9a317920c4938cf25716e414efc not found: ID does not exist"
Sep 10 23:13:50.676148 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.676124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27529e5a-96d1-40c6-9677-5b1776287655-tls-cert\") pod \"27529e5a-96d1-40c6-9677-5b1776287655\" (UID: \"27529e5a-96d1-40c6-9677-5b1776287655\") "
Sep 10 23:13:50.676291 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.676179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfkrr\" (UniqueName: \"kubernetes.io/projected/27529e5a-96d1-40c6-9677-5b1776287655-kube-api-access-kfkrr\") pod \"27529e5a-96d1-40c6-9677-5b1776287655\" (UID: \"27529e5a-96d1-40c6-9677-5b1776287655\") "
Sep 10 23:13:50.678357 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.678328    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27529e5a-96d1-40c6-9677-5b1776287655-kube-api-access-kfkrr" (OuterVolumeSpecName: "kube-api-access-kfkrr") pod "27529e5a-96d1-40c6-9677-5b1776287655" (UID: "27529e5a-96d1-40c6-9677-5b1776287655"). InnerVolumeSpecName "kube-api-access-kfkrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:13:50.686847 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.686820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27529e5a-96d1-40c6-9677-5b1776287655-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "27529e5a-96d1-40c6-9677-5b1776287655" (UID: "27529e5a-96d1-40c6-9677-5b1776287655"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:13:50.776996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.776967    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27529e5a-96d1-40c6-9677-5b1776287655-tls-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:13:50.776996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.776995    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfkrr\" (UniqueName: \"kubernetes.io/projected/27529e5a-96d1-40c6-9677-5b1776287655-kube-api-access-kfkrr\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:13:50.839934 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.839905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:50.925061 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.924959    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75cf5b7ffc-sjxll"]
Sep 10 23:13:50.927506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.927483    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75cf5b7ffc-sjxll"]
Sep 10 23:13:50.929414 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:50.929389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:51.921268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:51.921236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:52.029653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:52.029623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:52.833931 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:13:52.833900    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27529e5a-96d1-40c6-9677-5b1776287655" path="/var/lib/kubelet/pods/27529e5a-96d1-40c6-9677-5b1776287655/volumes"
Sep 10 23:14:06.979113 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:06.979078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:07.085700 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.085670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:07.183661 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.183628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:07.323331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.323253    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"]
Sep 10 23:14:07.323650 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.323637    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27529e5a-96d1-40c6-9677-5b1776287655" containerName="authorino"
Sep 10 23:14:07.323699 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.323653    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27529e5a-96d1-40c6-9677-5b1776287655" containerName="authorino"
Sep 10 23:14:07.323734 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.323710    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27529e5a-96d1-40c6-9677-5b1776287655" containerName="authorino"
Sep 10 23:14:07.326697 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.326668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.326947 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.326918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:07.328760 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.328741    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6ce17e-openshift-default-dockercfg-l4clm\""
Sep 10 23:14:07.340138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.340115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"]
Sep 10 23:14:07.380930 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.380879    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"]
Sep 10 23:14:07.386289 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.386262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"]
Sep 10 23:14:07.386448 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.386369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.388687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.388661    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-679c94-openshift-default-dockercfg-p5mc7\""
Sep 10 23:14:07.414274 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.414238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:07.430048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.429579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-data\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.430048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.429828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgfk\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-kube-api-access-twgfk\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.430048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.429935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-envoy\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.430048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5cw\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-kube-api-access-tl5cw\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.430347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5252afc0-120c-4b0e-8414-369c39abd59b-istio-podinfo\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.430347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-certs\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.430347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-istio-token\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.430347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-envoy\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.430347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-certs\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.430347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-token\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.430698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47562c88-cde0-4b21-8198-0e98316ce5c9-istiod-ca-cert\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.430698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-credential-socket\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.430698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-podinfo\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.430698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-socket\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.430698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-data\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.430698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-credential-socket\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.430992 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5252afc0-120c-4b0e-8414-369c39abd59b-istiod-ca-cert\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.430992 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.430779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-socket\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.448829 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.448798    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"]
Sep 10 23:14:07.452867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.452849    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.455007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.454983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7025b9-openshift-default-dockercfg-ggjtt\""
Sep 10 23:14:07.463957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.463932    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"]
Sep 10 23:14:07.490500 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.490475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:07.531775 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.531750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsfg\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-kube-api-access-pnsfg\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.531912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.531784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-podinfo\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.531912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.531814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-certs\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.531912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.531848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-socket\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.531912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.531863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-data\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.531912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.531889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-credential-socket\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.532171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.531923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5252afc0-120c-4b0e-8414-369c39abd59b-istiod-ca-cert\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.532171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.531950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b55da4-111f-4033-841f-5de247dcad01-istiod-ca-cert\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.532171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.531983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-socket\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.532171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-istio-token\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.532171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b55da4-111f-4033-841f-5de247dcad01-istio-podinfo\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.532171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-socket\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.532171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-data\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.532171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgfk\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-kube-api-access-twgfk\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-envoy\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5cw\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-kube-api-access-tl5cw\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5252afc0-120c-4b0e-8414-369c39abd59b-istio-podinfo\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-certs\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-socket\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-credential-socket\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-istio-token\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-data\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-envoy\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-certs\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-data\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-token\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-socket\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47562c88-cde0-4b21-8198-0e98316ce5c9-istiod-ca-cert\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.532587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-envoy\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.533397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-credential-socket\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.533397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-credential-socket\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.533397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-certs\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.533397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-certs\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.533397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-credential-socket\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.533397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-data\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.533397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.532997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5252afc0-120c-4b0e-8414-369c39abd59b-istiod-ca-cert\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.535639 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.534090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47562c88-cde0-4b21-8198-0e98316ce5c9-istiod-ca-cert\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.535639 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.535564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-envoy\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.536449 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.536261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-envoy\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.536449 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.536406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-podinfo\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.536737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.536717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5252afc0-120c-4b0e-8414-369c39abd59b-istio-podinfo\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.545250 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.545220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5cw\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-kube-api-access-tl5cw\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.545592 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.545571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-istio-token\") pod \"e2e-inf-a-679c94-openshift-default-586587fdb6-727nf\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") " pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.546137 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.546094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-token\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.546520 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.546488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgfk\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-kube-api-access-twgfk\") pod \"e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") " pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.580087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.580017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:07.633570 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.633515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-data\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.633771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.633582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-envoy\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.633771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.633611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsfg\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-kube-api-access-pnsfg\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.633771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.633639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-certs\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.633771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.633691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b55da4-111f-4033-841f-5de247dcad01-istiod-ca-cert\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.633771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.633752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-istio-token\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.634043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.633782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b55da4-111f-4033-841f-5de247dcad01-istio-podinfo\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.634043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.633809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-socket\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.634043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.633875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-credential-socket\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.634043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.634001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-data\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.634258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.634105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-certs\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.634258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.634220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-credential-socket\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.634390 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.634336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-socket\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.634729 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.634676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b55da4-111f-4033-841f-5de247dcad01-istiod-ca-cert\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.636322 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.636295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-envoy\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.636402 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.636381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b55da4-111f-4033-841f-5de247dcad01-istio-podinfo\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.639250 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.639203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:07.641068 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.641048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-istio-token\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.641396 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.641376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsfg\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-kube-api-access-pnsfg\") pod \"e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") " pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.642692 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.642674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:07.699065 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.698811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:07.768280 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.768055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:07.929676 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.929647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"]
Sep 10 23:14:07.930282 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:14:07.930254    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47562c88_cde0_4b21_8198_0e98316ce5c9.slice/crio-cc2f696a2f70bafed2a27a0206a3d83432c8402d4a8799b2517deb34b6f02654 WatchSource:0}: Error finding container cc2f696a2f70bafed2a27a0206a3d83432c8402d4a8799b2517deb34b6f02654: Status 404 returned error can't find the container with id cc2f696a2f70bafed2a27a0206a3d83432c8402d4a8799b2517deb34b6f02654
Sep 10 23:14:07.932662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.932631    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:07.932739 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.932696    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:07.932739 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.932730    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:07.987236 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.987211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"]
Sep 10 23:14:07.991628 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:14:07.991603    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5252afc0_120c_4b0e_8414_369c39abd59b.slice/crio-89c22770ccd14e96289ea7079e5ecfa992536966aabfd5f4b2b2df04d1d034f6 WatchSource:0}: Error finding container 89c22770ccd14e96289ea7079e5ecfa992536966aabfd5f4b2b2df04d1d034f6: Status 404 returned error can't find the container with id 89c22770ccd14e96289ea7079e5ecfa992536966aabfd5f4b2b2df04d1d034f6
Sep 10 23:14:07.993919 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.993875    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:07.994010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.993965    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:07.994082 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:07.994031    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:08.050401 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.050374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"]
Sep 10 23:14:08.052317 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:14:08.052289    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b55da4_111f_4033_841f_5de247dcad01.slice/crio-b11250dd0a1d09f14537b12845197395c1b10cae42c69c96ebd86abb94cb6462 WatchSource:0}: Error finding container b11250dd0a1d09f14537b12845197395c1b10cae42c69c96ebd86abb94cb6462: Status 404 returned error can't find the container with id b11250dd0a1d09f14537b12845197395c1b10cae42c69c96ebd86abb94cb6462
Sep 10 23:14:08.055338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.055298    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:08.055442 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.055382    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:08.055442 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.055418    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:08.061586 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.061416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:08.312065 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.311976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:08.412434 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.412404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:08.701443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.701410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" event={"ID":"47562c88-cde0-4b21-8198-0e98316ce5c9","Type":"ContainerStarted","Data":"35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1"}
Sep 10 23:14:08.701698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.701471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" event={"ID":"47562c88-cde0-4b21-8198-0e98316ce5c9","Type":"ContainerStarted","Data":"cc2f696a2f70bafed2a27a0206a3d83432c8402d4a8799b2517deb34b6f02654"}
Sep 10 23:14:08.703069 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.703046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" event={"ID":"78b55da4-111f-4033-841f-5de247dcad01","Type":"ContainerStarted","Data":"92a33e98469af3d5e7930aa791d09b95e5c206ab3f9aa7063c73e249c34c469b"}
Sep 10 23:14:08.703215 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.703086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" event={"ID":"78b55da4-111f-4033-841f-5de247dcad01","Type":"ContainerStarted","Data":"b11250dd0a1d09f14537b12845197395c1b10cae42c69c96ebd86abb94cb6462"}
Sep 10 23:14:08.704800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.704778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" event={"ID":"5252afc0-120c-4b0e-8414-369c39abd59b","Type":"ContainerStarted","Data":"7478dfb69d4774b6480d01f1fb37e1ac6d25cd01b1230eab6195f602c18f01fa"}
Sep 10 23:14:08.704898 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.704819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" event={"ID":"5252afc0-120c-4b0e-8414-369c39abd59b","Type":"ContainerStarted","Data":"89c22770ccd14e96289ea7079e5ecfa992536966aabfd5f4b2b2df04d1d034f6"}
Sep 10 23:14:08.720742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.720692    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podStartSLOduration=1.720676512 podStartE2EDuration="1.720676512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:08.719963674 +0000 UTC m=+748.421596630" watchObservedRunningTime="2026-09-10 23:14:08.720676512 +0000 UTC m=+748.422309456"
Sep 10 23:14:08.741263 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.741218    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podStartSLOduration=1.7412053699999999 podStartE2EDuration="1.74120537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:08.739506915 +0000 UTC m=+748.441139857" watchObservedRunningTime="2026-09-10 23:14:08.74120537 +0000 UTC m=+748.442838313"
Sep 10 23:14:08.759929 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.759889    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podStartSLOduration=1.759874148 podStartE2EDuration="1.759874148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:08.758859924 +0000 UTC m=+748.460492867" watchObservedRunningTime="2026-09-10 23:14:08.759874148 +0000 UTC m=+748.461507093"
Sep 10 23:14:08.762389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.762368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:08.769445 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:08.769429    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:09.640024 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:09.639991    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:09.645113 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:09.645089    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:09.699292 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:09.699268    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:09.703837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:09.703816    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:09.710115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:09.710054    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:09.710226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:09.710212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:09.710828 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:09.710805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:09.710920 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:09.710855    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:09.774508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:09.774479    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:10.661508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:10.661448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:10.717165 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:10.717128    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:10.717988 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:10.717969    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:10.748601 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:10.748563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:10.819290 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:10.819255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:10.888642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:10.888608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:14:10.891002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:10.890979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:11.897281 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:11.897246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:11.967168 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:11.967132    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:12.115581 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:12.115523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:12.501315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:12.501271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:12.592698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:12.592669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:12.914686 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:12.914643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:12.982877 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:12.982841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:13.046722 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:13.046691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:13.519303 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:13.519273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:13.620090 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:13.620058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:14.090677 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:14.090638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:14.189024 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:14.188996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:14.313036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:14.313009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:14.693972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:14.693943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:14.794465 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:14.794430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:15.200923 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:15.200892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:15.563834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:15.563738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:15.930933 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:15.930884    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"]
Sep 10 23:14:15.935693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:15.935673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:15.938090 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:15.938071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-689fe3-openshift-default-dockercfg-dtmls\""
Sep 10 23:14:15.947556 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:15.947522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"]
Sep 10 23:14:15.960469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:15.960448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:16.016479 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.016449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-token\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.016641 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.016492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-envoy\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.016641 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.016603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-credential-socket\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.016641 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.016638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-socket\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.016773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.016654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istiod-ca-cert\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.016773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.016700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-podinfo\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.016773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.016719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-certs\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.016773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.016757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-data\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.016910 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.016797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pszx4\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-kube-api-access-pszx4\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.071512 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.071476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:16.117763 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.117734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pszx4\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-kube-api-access-pszx4\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.117916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.117783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-token\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.117916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.117820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-envoy\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.117916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.117855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-credential-socket\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.117916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.117900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-socket\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.118673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.118046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istiod-ca-cert\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.118673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.118077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-podinfo\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.118673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.118104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-certs\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.118673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.118141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-data\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.118673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.118591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-credential-socket\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.118673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.118616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-certs\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.118673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.118630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-socket\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.119238 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.118768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-data\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.119238 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.119124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istiod-ca-cert\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.120773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.120753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-envoy\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.120987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.120970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-podinfo\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.125928 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.125905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-token\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.126017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.125990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pszx4\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-kube-api-access-pszx4\") pod \"e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") " pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.171487 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.171461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:16.247596 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.247492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:16.290774 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.290745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:16.527067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.527040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"]
Sep 10 23:14:16.529032 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:14:16.528993    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bedfa5d_56c4_43eb_95a2_8c83f1e28b75.slice/crio-ec8186850c62aee9249e4f19d664975acfdf501397df06853a50905964bd8aea WatchSource:0}: Error finding container ec8186850c62aee9249e4f19d664975acfdf501397df06853a50905964bd8aea: Status 404 returned error can't find the container with id ec8186850c62aee9249e4f19d664975acfdf501397df06853a50905964bd8aea
Sep 10 23:14:16.531933 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.531900    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:16.532022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.531966    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:16.532022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.532000    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:16.560120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.560092    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"]
Sep 10 23:14:16.565226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.565191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.567757 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.567734    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f897c8bb-openshift-default-dockercfg-sxvmv\""
Sep 10 23:14:16.576498 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.576471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"]
Sep 10 23:14:16.585287 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.585255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:16.623994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.623968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-podinfo\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.624129 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.624046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-certs\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.624129 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.624079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-data\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.624129 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.624096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-token\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.624129 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.624116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istiod-ca-cert\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.624338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.624205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-socket\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.624338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.624234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-envoy\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.624338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.624266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrp4x\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-kube-api-access-wrp4x\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.624338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.624313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-credential-socket\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.725917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.725873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-credential-socket\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.726089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.725964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-podinfo\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.726089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.726038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-certs\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.726089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.726082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-data\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.726385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.726108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-token\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.726385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.726136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istiod-ca-cert\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.726385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.726194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-socket\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.726385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.726224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-envoy\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.726385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.726270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrp4x\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-kube-api-access-wrp4x\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.727034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.727003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-credential-socket\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.727882 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.727851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-certs\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.728121 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.728098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-data\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.728368 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.728342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-socket\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.729011 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.728980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istiod-ca-cert\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.735562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.733486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-envoy\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.735562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.734006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-podinfo\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.737754 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.737709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrp4x\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-kube-api-access-wrp4x\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.739567 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.739170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-token\") pod \"e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") " pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:16.761192 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.760272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" event={"ID":"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75","Type":"ContainerStarted","Data":"d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3"}
Sep 10 23:14:16.761192 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.760339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" event={"ID":"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75","Type":"ContainerStarted","Data":"ec8186850c62aee9249e4f19d664975acfdf501397df06853a50905964bd8aea"}
Sep 10 23:14:16.784154 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.784012    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" podStartSLOduration=1.783990456 podStartE2EDuration="1.783990456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:16.781701374 +0000 UTC m=+756.483334315" watchObservedRunningTime="2026-09-10 23:14:16.783990456 +0000 UTC m=+756.485623400"
Sep 10 23:14:16.889789 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:16.889751    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:17.014717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.014684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:17.184506 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:14:17.184473    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd97e3b_f4e8_49b0_8cae_8485019f582a.slice/crio-8a85c89308400c48e9970a0a37dc9881f396020e9ec53e2d07ca61f1dbff99f6 WatchSource:0}: Error finding container 8a85c89308400c48e9970a0a37dc9881f396020e9ec53e2d07ca61f1dbff99f6: Status 404 returned error can't find the container with id 8a85c89308400c48e9970a0a37dc9881f396020e9ec53e2d07ca61f1dbff99f6
Sep 10 23:14:17.186295 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.186263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"]
Sep 10 23:14:17.187150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.187116    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:17.187265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.187191    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:17.187265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.187235    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:17.248565 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.248514    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:17.253871 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.253847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:17.609890 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.609857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:17.767321 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.767288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" event={"ID":"6cd97e3b-f4e8-49b0-8cae-8485019f582a","Type":"ContainerStarted","Data":"dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0"}
Sep 10 23:14:17.767486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.767340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" event={"ID":"6cd97e3b-f4e8-49b0-8cae-8485019f582a","Type":"ContainerStarted","Data":"8a85c89308400c48e9970a0a37dc9881f396020e9ec53e2d07ca61f1dbff99f6"}
Sep 10 23:14:17.767692 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.767676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:17.768717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.768701    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:17.788133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.788083    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" podStartSLOduration=1.78806613 podStartE2EDuration="1.78806613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:17.787632269 +0000 UTC m=+757.489265212" watchObservedRunningTime="2026-09-10 23:14:17.78806613 +0000 UTC m=+757.489699074"
Sep 10 23:14:17.890280 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:17.890196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:18.682680 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:18.682644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:18.770889 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:18.770854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:18.895822 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:18.895793    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:19.270123 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:19.270075    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:19.364218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:19.364179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:19.781511 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:19.781259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:19.782637 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:19.782531    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:19.848570 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:19.843337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:19.946622 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:19.946314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:20.463387 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:20.463357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:20.579899 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:20.579852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:20.662175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:20.662145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:20.952096 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:20.952049    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"]
Sep 10 23:14:20.973116 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:20.973087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:20.977136 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:20.977113    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f897c8bb-serving-cert\""
Sep 10 23:14:20.997390 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:20.996114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:14:20.997390 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:20.996425    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5kfc5\""
Sep 10 23:14:20.997390 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:20.996716    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:14:21.001828 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.001803    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:14:21.009046 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.008986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:21.070620 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.004579    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"]
Sep 10 23:14:21.092911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.092872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8w8\" (UniqueName: \"kubernetes.io/projected/45c38870-19eb-45d9-b5cd-1c9189ef03fa-kube-api-access-zq8w8\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.093139 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.093120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.093307 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.093292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c38870-19eb-45d9-b5cd-1c9189ef03fa-openshift-service-ca\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.093425 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.093412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-metrics-certs\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.143593 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.140403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:21.195189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.195137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8w8\" (UniqueName: \"kubernetes.io/projected/45c38870-19eb-45d9-b5cd-1c9189ef03fa-kube-api-access-zq8w8\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.195402 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.195207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.195402 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.195259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c38870-19eb-45d9-b5cd-1c9189ef03fa-openshift-service-ca\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.195402 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.195284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-metrics-certs\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.195730 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:14:21.195709    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-serving-cert: secret "maas-api-e2e-ait-f897c8bb-serving-cert" not found
Sep 10 23:14:21.196838 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:14:21.195892    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls podName:45c38870-19eb-45d9-b5cd-1c9189ef03fa nodeName:}" failed. No retries permitted until 2026-09-10 23:14:21.695868002 +0000 UTC m=+761.397500940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls") pod "maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl" (UID: "45c38870-19eb-45d9-b5cd-1c9189ef03fa") : secret "maas-api-e2e-ait-f897c8bb-serving-cert" not found
Sep 10 23:14:21.196838 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.196792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c38870-19eb-45d9-b5cd-1c9189ef03fa-openshift-service-ca\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.210630 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.200053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-metrics-certs\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.214789 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.214622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8w8\" (UniqueName: \"kubernetes.io/projected/45c38870-19eb-45d9-b5cd-1c9189ef03fa-kube-api-access-zq8w8\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.659565 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.659180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:21.705562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.705161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:21.705562 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:14:21.705426    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-serving-cert: secret "maas-api-e2e-ait-f897c8bb-serving-cert" not found
Sep 10 23:14:21.705562 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:14:21.705506    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls podName:45c38870-19eb-45d9-b5cd-1c9189ef03fa nodeName:}" failed. No retries permitted until 2026-09-10 23:14:22.705485095 +0000 UTC m=+762.407118031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls") pod "maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl" (UID: "45c38870-19eb-45d9-b5cd-1c9189ef03fa") : secret "maas-api-e2e-ait-f897c8bb-serving-cert" not found
Sep 10 23:14:21.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:21.744946    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:22.075655 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:22.075442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:22.287816 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:22.287743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:22.728611 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:22.721254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:22.749569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:22.740171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls\") pod \"maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:22.888994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:22.888950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:22.897326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:22.897288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:23.326629 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:23.324186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"]
Sep 10 23:14:23.341320 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:14:23.341264    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c38870_19eb_45d9_b5cd_1c9189ef03fa.slice/crio-305e4c6fa221850ce07b8d3c69b442616fd15c255a513780bc4e0c2be92d51ba WatchSource:0}: Error finding container 305e4c6fa221850ce07b8d3c69b442616fd15c255a513780bc4e0c2be92d51ba: Status 404 returned error can't find the container with id 305e4c6fa221850ce07b8d3c69b442616fd15c255a513780bc4e0c2be92d51ba
Sep 10 23:14:23.854187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:23.854149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl" event={"ID":"45c38870-19eb-45d9-b5cd-1c9189ef03fa","Type":"ContainerStarted","Data":"305e4c6fa221850ce07b8d3c69b442616fd15c255a513780bc4e0c2be92d51ba"}
Sep 10 23:14:25.686333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:25.686260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:25.837337 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:25.837306    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:25.941210 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:25.941064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:26.036269 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:26.036193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:27.078981 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:27.078946    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:28.692762 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:28.692726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:28.908320 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:28.908284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl" event={"ID":"45c38870-19eb-45d9-b5cd-1c9189ef03fa","Type":"ContainerStarted","Data":"3b0130cd3d7dbfda2bbaf6a28af16ed9ed457ef0ce9278c0f7a08bc903206433"}
Sep 10 23:14:28.908511 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:28.908391    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:28.931263 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:28.931191    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl" podStartSLOduration=3.957483753 podStartE2EDuration="8.931167599s" totalImagesPullingTime="4.973683846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:28.927268758 +0000 UTC m=+768.628901702" watchObservedRunningTime="2026-09-10 23:14:28.931167599 +0000 UTC m=+768.632800554"
Sep 10 23:14:30.277381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:30.277342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:30.996211 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:30.779330    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:14:32.942087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.942043    2580 generic.go:356] "Generic (PLEG): container finished" podID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerID="35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1" exitCode=0
Sep 10 23:14:32.942608 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.942117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" event={"ID":"47562c88-cde0-4b21-8198-0e98316ce5c9","Type":"ContainerDied","Data":"35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1"}
Sep 10 23:14:32.944137 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.944108    2580 scope.go:117] "RemoveContainer" containerID="35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1"
Sep 10 23:14:32.944703 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.944672    2580 generic.go:356] "Generic (PLEG): container finished" podID="78b55da4-111f-4033-841f-5de247dcad01" containerID="92a33e98469af3d5e7930aa791d09b95e5c206ab3f9aa7063c73e249c34c469b" exitCode=0
Sep 10 23:14:32.944813 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.944710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" event={"ID":"78b55da4-111f-4033-841f-5de247dcad01","Type":"ContainerDied","Data":"92a33e98469af3d5e7930aa791d09b95e5c206ab3f9aa7063c73e249c34c469b"}
Sep 10 23:14:32.945620 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.945587    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:32.945718 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.945673    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:32.945783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.945717    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:32.946658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.946271    2580 scope.go:117] "RemoveContainer" containerID="92a33e98469af3d5e7930aa791d09b95e5c206ab3f9aa7063c73e249c34c469b"
Sep 10 23:14:32.947249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.947193    2580 generic.go:356] "Generic (PLEG): container finished" podID="5252afc0-120c-4b0e-8414-369c39abd59b" containerID="7478dfb69d4774b6480d01f1fb37e1ac6d25cd01b1230eab6195f602c18f01fa" exitCode=0
Sep 10 23:14:32.947354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.947284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" event={"ID":"5252afc0-120c-4b0e-8414-369c39abd59b","Type":"ContainerDied","Data":"7478dfb69d4774b6480d01f1fb37e1ac6d25cd01b1230eab6195f602c18f01fa"}
Sep 10 23:14:32.947689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.947674    2580 scope.go:117] "RemoveContainer" containerID="7478dfb69d4774b6480d01f1fb37e1ac6d25cd01b1230eab6195f602c18f01fa"
Sep 10 23:14:32.947749 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.947672    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:32.947810 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.947788    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:32.947858 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.947842    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:32.948808 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.948779    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:32.948870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.948856    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:32.948918 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.948906    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:32.968474 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:32.968444    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:33.955873 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:33.955845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" event={"ID":"47562c88-cde0-4b21-8198-0e98316ce5c9","Type":"ContainerStarted","Data":"7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f"}
Sep 10 23:14:33.958526 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:33.958504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" event={"ID":"78b55da4-111f-4033-841f-5de247dcad01","Type":"ContainerStarted","Data":"3ea29df5541e0588cfbce0197722e009c0c5a04174e20fb5f0a9ba5895afbc42"}
Sep 10 23:14:33.960769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:33.960751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" event={"ID":"5252afc0-120c-4b0e-8414-369c39abd59b","Type":"ContainerStarted","Data":"44f3444b2915b91de7124a1303a795e35693d4c637212de0d236749c9fb202f4"}
Sep 10 23:14:34.137583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.137526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:34.640146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.640104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:34.640362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.640257    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 23:14:34.640362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.640301    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 23:14:34.699423 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.699390    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:34.699909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.699878    2580 patch_prober.go:28] interesting pod/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 23:14:34.700066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.699954    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 23:14:34.768975 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.768926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:34.769691 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.769662    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 23:14:34.769891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.769870    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 23:14:34.891468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.891379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:34.932786 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:34.932743    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:14:35.553228 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:35.553186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:35.639674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:35.639641    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 23:14:35.639951 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:35.639919    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 23:14:35.656585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:35.656231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:35.699943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:35.699463    2580 patch_prober.go:28] interesting pod/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 23:14:35.699943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:35.699527    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 23:14:35.768913 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:35.768876    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 23:14:35.769126 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:35.768937    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 23:14:35.915393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:35.915351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:36.021244 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.021200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:36.121305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.121264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:36.208620 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.204247    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"]
Sep 10 23:14:36.212591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.210948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:36.217353 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.217329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.224995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.224958    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-568856-openshift-default-dockercfg-pjsd6\""
Sep 10 23:14:36.239558 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.239506    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"]
Sep 10 23:14:36.277333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.277274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:36.364301 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.364261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-certs\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.364649 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.364625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-istio-token\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.364819 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.364802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzwgb\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-kube-api-access-jzwgb\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.364937 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.364923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-socket\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.365092 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.365076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-envoy\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.365227 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.365214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74498af-163d-4e8c-b2ff-196463327948-istio-podinfo\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.365360 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.365347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-data\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.365569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.365521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-credential-socket\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.365695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.365681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74498af-163d-4e8c-b2ff-196463327948-istiod-ca-cert\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.467252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.467150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzwgb\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-kube-api-access-jzwgb\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.467487 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.467471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-socket\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.467659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.467644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-envoy\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.467799 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.467787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74498af-163d-4e8c-b2ff-196463327948-istio-podinfo\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.467921 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.467907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-data\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.468075 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.468055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-credential-socket\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.468178 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.468166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74498af-163d-4e8c-b2ff-196463327948-istiod-ca-cert\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.468296 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.468284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-certs\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.468401 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.468389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-istio-token\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.469450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.469415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-socket\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.471314 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.469853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-data\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.471447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.470899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-credential-socket\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.471528 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.471062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74498af-163d-4e8c-b2ff-196463327948-istiod-ca-cert\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.471633 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.471261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-certs\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.486565 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.485426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-envoy\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.490566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.487426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74498af-163d-4e8c-b2ff-196463327948-istio-podinfo\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.498159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.498119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-istio-token\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.501564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.498933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzwgb\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-kube-api-access-jzwgb\") pod \"e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") " pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.571420 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.571378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:36.614771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.614713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:36.641171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.641118    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 23:14:36.641476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.641443    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 23:14:36.701269 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.701231    2580 patch_prober.go:28] interesting pod/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 23:14:36.701531 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.701502    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 23:14:36.769745 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.769616    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 23:14:36.769745 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:36.769731    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 23:14:37.229568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.229261    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"]
Sep 10 23:14:37.243350 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:14:37.243297    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74498af_163d_4e8c_b2ff_196463327948.slice/crio-a06e92f23e4cd344dbc7a73080f3d5c9753f2528ef53d21dd6107fce81dd7ff3 WatchSource:0}: Error finding container a06e92f23e4cd344dbc7a73080f3d5c9753f2528ef53d21dd6107fce81dd7ff3: Status 404 returned error can't find the container with id a06e92f23e4cd344dbc7a73080f3d5c9753f2528ef53d21dd6107fce81dd7ff3
Sep 10 23:14:37.247475 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.246769    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:37.247475 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.246861    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:37.247475 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.246906    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:37.606930 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.606748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:37.640293 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.639868    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 23:14:37.640293 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.639930    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 23:14:37.700392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.699938    2580 patch_prober.go:28] interesting pod/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 23:14:37.700392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.700000    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 23:14:37.769692 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.769659    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 23:14:37.769991 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:37.769934    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 23:14:38.091625 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.091322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" event={"ID":"b74498af-163d-4e8c-b2ff-196463327948","Type":"ContainerStarted","Data":"e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b"}
Sep 10 23:14:38.091625 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.091392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" event={"ID":"b74498af-163d-4e8c-b2ff-196463327948","Type":"ContainerStarted","Data":"a06e92f23e4cd344dbc7a73080f3d5c9753f2528ef53d21dd6107fce81dd7ff3"}
Sep 10 23:14:38.551226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.551191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:38.572044 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.571997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:38.580027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.579998    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:38.602335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.602269    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" podStartSLOduration=2.6022447680000003 podStartE2EDuration="2.602244768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:38.125752 +0000 UTC m=+777.827384944" watchObservedRunningTime="2026-09-10 23:14:38.602244768 +0000 UTC m=+778.303877713"
Sep 10 23:14:38.611496 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.611461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:38.640757 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.640698    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 23:14:38.640971 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.640783    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 23:14:38.700237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.700193    2580 patch_prober.go:28] interesting pod/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 23:14:38.700445 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.700274    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 23:14:38.760152 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.760096    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:38.769664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.769612    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 23:14:38.769866 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:38.769693    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 23:14:39.104013 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.103978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:39.110073 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.110022    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:14:39.641473 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.641429    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 23:14:39.641966 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.641497    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 23:14:39.700565 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.700046    2580 patch_prober.go:28] interesting pod/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 23:14:39.700565 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.700120    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 23:14:39.710141 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.710090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:39.711689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.711659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:39.775568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.768951    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 23:14:39.775568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:39.769019    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 23:14:40.033873 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.033764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:40.138572 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.138502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:40.640098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.639637    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 23:14:40.640098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.639703    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 23:14:40.699895 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.699852    2580 patch_prober.go:28] interesting pod/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 23:14:40.700353 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.699918    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 23:14:40.717653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.716728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:40.764865 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.764823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:40.769386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.769341    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 23:14:40.769685 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:40.769636    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 23:14:41.068481 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.068392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:41.176140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.176050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:41.268406 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.268343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:41.368733 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.368689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:41.640693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.640605    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 23:14:41.641004 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.640965    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 23:14:41.700508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.700470    2580 patch_prober.go:28] interesting pod/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 23:14:41.701115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.701080    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 23:14:41.768945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.768911    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 23:14:41.769202 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:41.769175    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 23:14:42.037216 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.037092    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:42.173753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.172071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:42.606899 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.606817    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:42.655602 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.655558    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 23:14:42.655831 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.655645    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 23:14:42.706091 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.706054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:42.769667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.769630    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 23:14:42.769938 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.769903    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 23:14:42.944514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.944237    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:42.946692 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:42.946630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:14:43.007726 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:43.007689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:43.329362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:43.329255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:43.544785 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:43.544720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:43.644780 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:43.644749    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:43.645951 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:43.645929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:14:43.773960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:43.773931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:43.775167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:43.775146    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:14:43.920903 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:43.920815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:44.719359 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:44.719329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.208800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.208771    2580 generic.go:356] "Generic (PLEG): container finished" podID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerID="dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0" exitCode=0
Sep 10 23:14:45.209162 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.208856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" event={"ID":"6cd97e3b-f4e8-49b0-8cae-8485019f582a","Type":"ContainerDied","Data":"dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0"}
Sep 10 23:14:45.209453 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.209434    2580 scope.go:117] "RemoveContainer" containerID="dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0"
Sep 10 23:14:45.210894 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.210867    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:45.210962 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.210932    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:45.211005 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.210962    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:45.478837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.478737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.624867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.624829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.737098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.736853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.849898 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:45.849864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:46.001480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.001399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:46.216876 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.216835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" event={"ID":"6cd97e3b-f4e8-49b0-8cae-8485019f582a","Type":"ContainerStarted","Data":"17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef"}
Sep 10 23:14:46.219124 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.219094    2580 generic.go:356] "Generic (PLEG): container finished" podID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerID="d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3" exitCode=0
Sep 10 23:14:46.219335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.219213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" event={"ID":"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75","Type":"ContainerDied","Data":"d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3"}
Sep 10 23:14:46.219909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.219893    2580 scope.go:117] "RemoveContainer" containerID="d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3"
Sep 10 23:14:46.221521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.221485    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:46.221654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.221588    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:46.221654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.221637    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:14:46.890428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.890395    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:46.891659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.891634    2580 patch_prober.go:28] interesting pod/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 23:14:46.891796 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:46.891689    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 23:14:47.232422 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:47.232328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" event={"ID":"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75","Type":"ContainerStarted","Data":"2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697"}
Sep 10 23:14:47.248221 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:47.248191    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:47.249471 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:47.249442    2580 patch_prober.go:28] interesting pod/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 23:14:47.249600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:47.249496    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 23:14:47.767909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:47.767870    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:47.890374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:47.890343    2580 patch_prober.go:28] interesting pod/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 23:14:47.890566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:47.890400    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 23:14:48.248657 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:48.248619    2580 patch_prober.go:28] interesting pod/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 23:14:48.249217 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:48.249185    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 23:14:48.891079 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:48.891045    2580 patch_prober.go:28] interesting pod/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 23:14:48.891257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:48.891104    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 23:14:49.251431 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:49.251330    2580 patch_prober.go:28] interesting pod/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 23:14:49.251431 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:49.251393    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 23:14:49.781444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:49.781406    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:49.890331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:49.890298    2580 patch_prober.go:28] interesting pod/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 23:14:49.890511 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:49.890355    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 23:14:50.248701 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:50.248664    2580 patch_prober.go:28] interesting pod/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 23:14:50.249009 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:50.248976    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 23:14:50.890895 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:50.890867    2580 patch_prober.go:28] interesting pod/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 23:14:50.891290 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:50.890927    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 23:14:51.248363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:51.248268    2580 patch_prober.go:28] interesting pod/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 23:14:51.248363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:51.248340    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 23:14:51.894773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:51.894749    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:51.895907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:51.895886    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:14:51.965413 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:51.965380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:52.023647 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:52.023605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:52.248897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:52.248807    2580 patch_prober.go:28] interesting pod/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 23:14:52.248897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:52.248862    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 23:14:53.110395 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:53.110362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:14:53.252489 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:53.252458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:14:53.253549 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:14:53.253520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:15:00.203037 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.203002    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb"]
Sep 10 23:15:00.205874 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.205858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb"
Sep 10 23:15:00.209806 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.209782    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:15:00.210140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.210121    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq"]
Sep 10 23:15:00.212842 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.212665    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq"
Sep 10 23:15:00.223585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.223534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb"
Sep 10 23:15:00.224885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.224865    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb"]
Sep 10 23:15:00.230624 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.230599    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6"]
Sep 10 23:15:00.250308 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.244703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq"
Sep 10 23:15:00.250308 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.245007    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq"]
Sep 10 23:15:00.250308 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.245120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6"
Sep 10 23:15:00.250308 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.249029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6"]
Sep 10 23:15:00.290023 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.289999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6"
Sep 10 23:15:00.536988 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.536959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb"]
Sep 10 23:15:00.537403 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:15:00.537379    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e31206_e26b_4e21_b80c_1dda84dfd7dd.slice/crio-f9aaee3b0b0dd121f5a1d2faf35c1e084bd2a0364407c851823d4fc033db005a WatchSource:0}: Error finding container f9aaee3b0b0dd121f5a1d2faf35c1e084bd2a0364407c851823d4fc033db005a: Status 404 returned error can't find the container with id f9aaee3b0b0dd121f5a1d2faf35c1e084bd2a0364407c851823d4fc033db005a
Sep 10 23:15:00.546759 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.546732    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq"]
Sep 10 23:15:00.547865 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:15:00.547837    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce8a510_0a87_494d_8980_08c331f3736b.slice/crio-9b4bfd9da590e034b5c77ec7309832cf4e36fb28e961bcd95de86528cf0eea96 WatchSource:0}: Error finding container 9b4bfd9da590e034b5c77ec7309832cf4e36fb28e961bcd95de86528cf0eea96: Status 404 returned error can't find the container with id 9b4bfd9da590e034b5c77ec7309832cf4e36fb28e961bcd95de86528cf0eea96
Sep 10 23:15:00.589559 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:00.589511    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6"]
Sep 10 23:15:00.591773 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:15:00.591740    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250a7699_a794_4ddc_a08f_676ceea369a5.slice/crio-c354970f9692faa3e17bb43def92fda3fc2551a51d4ef7786c32a334e9728e28 WatchSource:0}: Error finding container c354970f9692faa3e17bb43def92fda3fc2551a51d4ef7786c32a334e9728e28: Status 404 returned error can't find the container with id c354970f9692faa3e17bb43def92fda3fc2551a51d4ef7786c32a334e9728e28
Sep 10 23:15:01.093507 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.093476    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"]
Sep 10 23:15:01.093769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.093745    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl" podUID="45c38870-19eb-45d9-b5cd-1c9189ef03fa" containerName="maas-api" containerID="cri-o://3b0130cd3d7dbfda2bbaf6a28af16ed9ed457ef0ce9278c0f7a08bc903206433" gracePeriod=30
Sep 10 23:15:01.137994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.137945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:01.204800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.204761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:01.304511 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.304478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:01.338789 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.338761    2580 generic.go:356] "Generic (PLEG): container finished" podID="45c38870-19eb-45d9-b5cd-1c9189ef03fa" containerID="3b0130cd3d7dbfda2bbaf6a28af16ed9ed457ef0ce9278c0f7a08bc903206433" exitCode=0
Sep 10 23:15:01.338951 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.338843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl" event={"ID":"45c38870-19eb-45d9-b5cd-1c9189ef03fa","Type":"ContainerDied","Data":"3b0130cd3d7dbfda2bbaf6a28af16ed9ed457ef0ce9278c0f7a08bc903206433"}
Sep 10 23:15:01.340316 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.340286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb" event={"ID":"e5e31206-e26b-4e21-b80c-1dda84dfd7dd","Type":"ContainerStarted","Data":"f9aaee3b0b0dd121f5a1d2faf35c1e084bd2a0364407c851823d4fc033db005a"}
Sep 10 23:15:01.342027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.341958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6" event={"ID":"250a7699-a794-4ddc-a08f-676ceea369a5","Type":"ContainerStarted","Data":"c354970f9692faa3e17bb43def92fda3fc2551a51d4ef7786c32a334e9728e28"}
Sep 10 23:15:01.343375 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.343352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq" event={"ID":"4ce8a510-0a87-494d-8980-08c331f3736b","Type":"ContainerStarted","Data":"9b4bfd9da590e034b5c77ec7309832cf4e36fb28e961bcd95de86528cf0eea96"}
Sep 10 23:15:01.357149 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.357128    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:15:01.365765 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.365741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:01.405574 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.405532    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq8w8\" (UniqueName: \"kubernetes.io/projected/45c38870-19eb-45d9-b5cd-1c9189ef03fa-kube-api-access-zq8w8\") pod \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") "
Sep 10 23:15:01.405741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.405605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-metrics-certs\") pod \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") "
Sep 10 23:15:01.405741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.405649    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c38870-19eb-45d9-b5cd-1c9189ef03fa-openshift-service-ca\") pod \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") "
Sep 10 23:15:01.405741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.405707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls\") pod \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\" (UID: \"45c38870-19eb-45d9-b5cd-1c9189ef03fa\") "
Sep 10 23:15:01.406066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.406043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c38870-19eb-45d9-b5cd-1c9189ef03fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45c38870-19eb-45d9-b5cd-1c9189ef03fa" (UID: "45c38870-19eb-45d9-b5cd-1c9189ef03fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:15:01.408151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.408112    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45c38870-19eb-45d9-b5cd-1c9189ef03fa" (UID: "45c38870-19eb-45d9-b5cd-1c9189ef03fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:15:01.408151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.408131    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "45c38870-19eb-45d9-b5cd-1c9189ef03fa" (UID: "45c38870-19eb-45d9-b5cd-1c9189ef03fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:15:01.408318 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.408252    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c38870-19eb-45d9-b5cd-1c9189ef03fa-kube-api-access-zq8w8" (OuterVolumeSpecName: "kube-api-access-zq8w8") pod "45c38870-19eb-45d9-b5cd-1c9189ef03fa" (UID: "45c38870-19eb-45d9-b5cd-1c9189ef03fa"). InnerVolumeSpecName "kube-api-access-zq8w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:01.453284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.453241    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"]
Sep 10 23:15:01.454014 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.453809    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy" containerID="cri-o://17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef" gracePeriod=30
Sep 10 23:15:01.486870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.486837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:01.507614 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.507498    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq8w8\" (UniqueName: \"kubernetes.io/projected/45c38870-19eb-45d9-b5cd-1c9189ef03fa-kube-api-access-zq8w8\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:01.507614 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.507531    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:01.507614 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.507575    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c38870-19eb-45d9-b5cd-1c9189ef03fa-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:01.507614 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.507588    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c38870-19eb-45d9-b5cd-1c9189ef03fa-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:01.938964 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:01.938906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:02.036485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.036456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:02.143022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.142973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:02.356672 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.356640    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"
Sep 10 23:15:02.359571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.359520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl" event={"ID":"45c38870-19eb-45d9-b5cd-1c9189ef03fa","Type":"ContainerDied","Data":"305e4c6fa221850ce07b8d3c69b442616fd15c255a513780bc4e0c2be92d51ba"}
Sep 10 23:15:02.359716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.359611    2580 scope.go:117] "RemoveContainer" containerID="3b0130cd3d7dbfda2bbaf6a28af16ed9ed457ef0ce9278c0f7a08bc903206433"
Sep 10 23:15:02.390983 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.390958    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"]
Sep 10 23:15:02.405602 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.405571    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f897c8bb-6dd7d69675-hzqsl"]
Sep 10 23:15:02.418892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.416787    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"]
Sep 10 23:15:02.418892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.417446    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c38870-19eb-45d9-b5cd-1c9189ef03fa" containerName="maas-api"
Sep 10 23:15:02.418892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.417462    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c38870-19eb-45d9-b5cd-1c9189ef03fa" containerName="maas-api"
Sep 10 23:15:02.418892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.417595    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c38870-19eb-45d9-b5cd-1c9189ef03fa" containerName="maas-api"
Sep 10 23:15:02.430325 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.422761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.430325 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.430008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"]
Sep 10 23:15:02.430325 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.430114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7dbe617a-gw-openshift-default-dockercfg-pw55g\""
Sep 10 23:15:02.440997 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.440357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:02.523610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.522100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr74q\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-kube-api-access-qr74q\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.523610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.522194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-credential-socket\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.523610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.522226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-certs\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.523610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.522293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-envoy\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.523610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.522359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-data\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.523610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.522401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-istio-token\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.523610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.522452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-socket\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.523610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.522488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a444014-619e-4d93-90c6-5942f593164d-istiod-ca-cert\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.523610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.522524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a444014-619e-4d93-90c6-5942f593164d-istio-podinfo\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.623728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.623429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-credential-socket\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.623728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.623473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-certs\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.623728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.623571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-envoy\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.623728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.623636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-data\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.623728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.623680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-istio-token\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.623728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.623726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-socket\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.624133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.623749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a444014-619e-4d93-90c6-5942f593164d-istiod-ca-cert\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.624133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.623780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a444014-619e-4d93-90c6-5942f593164d-istio-podinfo\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.624133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.623816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr74q\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-kube-api-access-qr74q\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.624916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.624487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-credential-socket\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.624916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.624842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-certs\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.625364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.625310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a444014-619e-4d93-90c6-5942f593164d-istiod-ca-cert\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.625527 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.625505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-socket\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.625527 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.625519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-data\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.628485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.627855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-envoy\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.628485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.628257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a444014-619e-4d93-90c6-5942f593164d-istio-podinfo\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.632853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.632829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr74q\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-kube-api-access-qr74q\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.633456 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.633435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-istio-token\") pod \"e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") " pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.744230 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.744196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:02.836242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.836209    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c38870-19eb-45d9-b5cd-1c9189ef03fa" path="/var/lib/kubelet/pods/45c38870-19eb-45d9-b5cd-1c9189ef03fa/volumes"
Sep 10 23:15:02.967425 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:02.967289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:03.029171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:03.029130    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:03.597883 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:03.595823    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"]
Sep 10 23:15:03.609073 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:03.609029    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:03.609168 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:03.609125    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:03.609223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:03.609168    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:04.388328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.388289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq" event={"ID":"4ce8a510-0a87-494d-8980-08c331f3736b","Type":"ContainerStarted","Data":"07e33d6fea479222d6dafd01a8ad8b6cac138e1dca70b90bff3b3987c4f843b7"}
Sep 10 23:15:04.389970 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.389943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" event={"ID":"4a444014-619e-4d93-90c6-5942f593164d","Type":"ContainerStarted","Data":"484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f"}
Sep 10 23:15:04.390109 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.389985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" event={"ID":"4a444014-619e-4d93-90c6-5942f593164d","Type":"ContainerStarted","Data":"c07ec9774309fab39d89329f68172808bb1ec6edc373f592390794b4b31f891a"}
Sep 10 23:15:04.391586 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.391564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb" event={"ID":"e5e31206-e26b-4e21-b80c-1dda84dfd7dd","Type":"ContainerStarted","Data":"c2bf9252a80392e27b732f1a28deb21f2c259c6f096469c6c7f2fe88295fa16b"}
Sep 10 23:15:04.393206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.393184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6" event={"ID":"250a7699-a794-4ddc-a08f-676ceea369a5","Type":"ContainerStarted","Data":"1d9f5420e08521070ebb259773001e6c92701849d7a9ea3ccc3e065979b09ace"}
Sep 10 23:15:04.394769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.394750    2580 generic.go:356] "Generic (PLEG): container finished" podID="b74498af-163d-4e8c-b2ff-196463327948" containerID="e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b" exitCode=0
Sep 10 23:15:04.394865 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.394779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" event={"ID":"b74498af-163d-4e8c-b2ff-196463327948","Type":"ContainerDied","Data":"e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b"}
Sep 10 23:15:04.395136 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.395121    2580 scope.go:117] "RemoveContainer" containerID="e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b"
Sep 10 23:15:04.396191 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.396160    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:04.396295 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.396244    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:04.396361 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.396294    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:04.403973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.403937    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq" podStartSLOduration=1.5547652360000002 podStartE2EDuration="4.403923791s" totalImagesPullingTime="2.849158555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:04.40276716 +0000 UTC m=+804.104400103" watchObservedRunningTime="2026-09-10 23:15:04.403923791 +0000 UTC m=+804.105556736"
Sep 10 23:15:04.435715 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.435668    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6" podStartSLOduration=1.6178237420000001 podStartE2EDuration="4.435651517s" totalImagesPullingTime="2.817827775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:04.434183156 +0000 UTC m=+804.135816102" watchObservedRunningTime="2026-09-10 23:15:04.435651517 +0000 UTC m=+804.137284472"
Sep 10 23:15:04.448628 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.448584    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb" podStartSLOduration=1.585947234 podStartE2EDuration="4.448566738s" totalImagesPullingTime="2.862619504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:04.447787666 +0000 UTC m=+804.149420611" watchObservedRunningTime="2026-09-10 23:15:04.448566738 +0000 UTC m=+804.150199677"
Sep 10 23:15:04.467906 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.467863    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" podStartSLOduration=2.467845345 podStartE2EDuration="2.467845345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:04.466068199 +0000 UTC m=+804.167701146" watchObservedRunningTime="2026-09-10 23:15:04.467845345 +0000 UTC m=+804.169478289"
Sep 10 23:15:04.745038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.745007    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:04.750000 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.749976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:04.967153 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:04.967067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:05.402328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.402295    2580 generic.go:356] "Generic (PLEG): container finished" podID="250a7699-a794-4ddc-a08f-676ceea369a5" containerID="1d9f5420e08521070ebb259773001e6c92701849d7a9ea3ccc3e065979b09ace" exitCode=0
Sep 10 23:15:05.402563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.402369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6" event={"ID":"250a7699-a794-4ddc-a08f-676ceea369a5","Type":"ContainerDied","Data":"1d9f5420e08521070ebb259773001e6c92701849d7a9ea3ccc3e065979b09ace"}
Sep 10 23:15:05.404356 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.404328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" event={"ID":"b74498af-163d-4e8c-b2ff-196463327948","Type":"ContainerStarted","Data":"30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f"}
Sep 10 23:15:05.406173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.406153    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ce8a510-0a87-494d-8980-08c331f3736b" containerID="07e33d6fea479222d6dafd01a8ad8b6cac138e1dca70b90bff3b3987c4f843b7" exitCode=0
Sep 10 23:15:05.406272 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.406234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq" event={"ID":"4ce8a510-0a87-494d-8980-08c331f3736b","Type":"ContainerDied","Data":"07e33d6fea479222d6dafd01a8ad8b6cac138e1dca70b90bff3b3987c4f843b7"}
Sep 10 23:15:05.407794 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.407764    2580 generic.go:356] "Generic (PLEG): container finished" podID="e5e31206-e26b-4e21-b80c-1dda84dfd7dd" containerID="c2bf9252a80392e27b732f1a28deb21f2c259c6f096469c6c7f2fe88295fa16b" exitCode=0
Sep 10 23:15:05.407887 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.407842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb" event={"ID":"e5e31206-e26b-4e21-b80c-1dda84dfd7dd","Type":"ContainerDied","Data":"c2bf9252a80392e27b732f1a28deb21f2c259c6f096469c6c7f2fe88295fa16b"}
Sep 10 23:15:05.408047 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.408034    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:05.409122 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.409102    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:05.572050 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.572016    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:15:05.573202 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.573177    2580 patch_prober.go:28] interesting pod/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 23:15:05.573306 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.573226    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 23:15:05.916052 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:05.916012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:06.016743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.016703    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:06.153894 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.153863    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"]
Sep 10 23:15:06.157431 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.157407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.161896 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.160829    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:15:06.161896 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.160843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5kfc5\""
Sep 10 23:15:06.161896 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.160842    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:15:06.161896 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.161032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7dbe617a-serving-cert\""
Sep 10 23:15:06.161896 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.161098    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:15:06.168498 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.168475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"]
Sep 10 23:15:06.182939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.182910    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:06.263563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.260608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7m79\" (UniqueName: \"kubernetes.io/projected/6d5cf26f-0df0-4f53-9f26-a060be81ed46-kube-api-access-j7m79\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.263563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.260812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-metrics-certs\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.263563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.260868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-maas-api-tls\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.263563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.261031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5cf26f-0df0-4f53-9f26-a060be81ed46-openshift-service-ca\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.283171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.283138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:06.361640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.361608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5cf26f-0df0-4f53-9f26-a060be81ed46-openshift-service-ca\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.361640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.361648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7m79\" (UniqueName: \"kubernetes.io/projected/6d5cf26f-0df0-4f53-9f26-a060be81ed46-kube-api-access-j7m79\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.361907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.361791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-metrics-certs\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.361907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.361860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-maas-api-tls\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.362020 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:15:06.362005    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-serving-cert: secret "maas-api-e2e-migrate-7dbe617a-serving-cert" not found
Sep 10 23:15:06.362073 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:15:06.362064    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-maas-api-tls podName:6d5cf26f-0df0-4f53-9f26-a060be81ed46 nodeName:}" failed. No retries permitted until 2026-09-10 23:15:06.862048927 +0000 UTC m=+806.563681848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-maas-api-tls") pod "maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh" (UID: "6d5cf26f-0df0-4f53-9f26-a060be81ed46") : secret "maas-api-e2e-migrate-7dbe617a-serving-cert" not found
Sep 10 23:15:06.362348 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.362327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5cf26f-0df0-4f53-9f26-a060be81ed46-openshift-service-ca\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.364383 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.364364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-metrics-certs\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.370391 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.370373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7m79\" (UniqueName: \"kubernetes.io/projected/6d5cf26f-0df0-4f53-9f26-a060be81ed46-kube-api-access-j7m79\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.571855 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.571826    2580 patch_prober.go:28] interesting pod/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 23:15:06.571985 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.571886    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 23:15:06.647568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.643851    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb"
Sep 10 23:15:06.682665 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.680943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:06.697078 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.696736    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6"
Sep 10 23:15:06.755270 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.754331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:06.765359 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.765269    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq"
Sep 10 23:15:06.856315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.856196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:06.859009 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.858986    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:15:06.873993 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.873967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-maas-api-tls\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.876409 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.876384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-maas-api-tls\") pod \"maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:06.974995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.974959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrp4x\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-kube-api-access-wrp4x\") pod \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") "
Sep 10 23:15:06.975468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-podinfo\") pod \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") "
Sep 10 23:15:06.975468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975079    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-data\") pod \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") "
Sep 10 23:15:06.975468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-envoy\") pod \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") "
Sep 10 23:15:06.975468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-certs\") pod \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") "
Sep 10 23:15:06.975468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975164    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-token\") pod \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") "
Sep 10 23:15:06.975468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istiod-ca-cert\") pod \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") "
Sep 10 23:15:06.975468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-socket\") pod \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") "
Sep 10 23:15:06.975468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-credential-socket\") pod \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\" (UID: \"6cd97e3b-f4e8-49b0-8cae-8485019f582a\") "
Sep 10 23:15:06.975982 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6cd97e3b-f4e8-49b0-8cae-8485019f582a" (UID: "6cd97e3b-f4e8-49b0-8cae-8485019f582a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:06.975982 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975756    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6cd97e3b-f4e8-49b0-8cae-8485019f582a" (UID: "6cd97e3b-f4e8-49b0-8cae-8485019f582a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:06.975982 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.975798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-data" (OuterVolumeSpecName: "istio-data") pod "6cd97e3b-f4e8-49b0-8cae-8485019f582a" (UID: "6cd97e3b-f4e8-49b0-8cae-8485019f582a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:06.976130 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.976034    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6cd97e3b-f4e8-49b0-8cae-8485019f582a" (UID: "6cd97e3b-f4e8-49b0-8cae-8485019f582a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:06.976531 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.976494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6cd97e3b-f4e8-49b0-8cae-8485019f582a" (UID: "6cd97e3b-f4e8-49b0-8cae-8485019f582a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:15:06.977735 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.977714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6cd97e3b-f4e8-49b0-8cae-8485019f582a" (UID: "6cd97e3b-f4e8-49b0-8cae-8485019f582a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:15:06.977831 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.977794    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-token" (OuterVolumeSpecName: "istio-token") pod "6cd97e3b-f4e8-49b0-8cae-8485019f582a" (UID: "6cd97e3b-f4e8-49b0-8cae-8485019f582a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:06.977923 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.977907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-kube-api-access-wrp4x" (OuterVolumeSpecName: "kube-api-access-wrp4x") pod "6cd97e3b-f4e8-49b0-8cae-8485019f582a" (UID: "6cd97e3b-f4e8-49b0-8cae-8485019f582a"). InnerVolumeSpecName "kube-api-access-wrp4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:06.978030 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:06.978015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6cd97e3b-f4e8-49b0-8cae-8485019f582a" (UID: "6cd97e3b-f4e8-49b0-8cae-8485019f582a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:07.072811 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.072726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:07.076348 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.076324    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:07.076348 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.076348    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:07.076494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.076359    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrp4x\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-kube-api-access-wrp4x\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:07.076494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.076367    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:07.076494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.076375    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:07.076494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.076383    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:07.076494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.076390    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd97e3b-f4e8-49b0-8cae-8485019f582a-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:07.076494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.076398    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:07.076494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.076405    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd97e3b-f4e8-49b0-8cae-8485019f582a-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:07.363383 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.363356    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"]
Sep 10 23:15:07.364649 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:15:07.364610    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5cf26f_0df0_4f53_9f26_a060be81ed46.slice/crio-f23402f4177464be4f9846bcdb21859edfabf0d8486987e4099f07d8c7508646 WatchSource:0}: Error finding container f23402f4177464be4f9846bcdb21859edfabf0d8486987e4099f07d8c7508646: Status 404 returned error can't find the container with id f23402f4177464be4f9846bcdb21859edfabf0d8486987e4099f07d8c7508646
Sep 10 23:15:07.427806 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.427762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh" event={"ID":"6d5cf26f-0df0-4f53-9f26-a060be81ed46","Type":"ContainerStarted","Data":"f23402f4177464be4f9846bcdb21859edfabf0d8486987e4099f07d8c7508646"}
Sep 10 23:15:07.429532 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.429501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6" event={"ID":"250a7699-a794-4ddc-a08f-676ceea369a5","Type":"ContainerDied","Data":"c354970f9692faa3e17bb43def92fda3fc2551a51d4ef7786c32a334e9728e28"}
Sep 10 23:15:07.429697 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.429570    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c354970f9692faa3e17bb43def92fda3fc2551a51d4ef7786c32a334e9728e28"
Sep 10 23:15:07.429697 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.429644    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6"
Sep 10 23:15:07.431996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.431966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq" event={"ID":"4ce8a510-0a87-494d-8980-08c331f3736b","Type":"ContainerDied","Data":"9b4bfd9da590e034b5c77ec7309832cf4e36fb28e961bcd95de86528cf0eea96"}
Sep 10 23:15:07.432120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.432015    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b4bfd9da590e034b5c77ec7309832cf4e36fb28e961bcd95de86528cf0eea96"
Sep 10 23:15:07.432120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.432084    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq"
Sep 10 23:15:07.435022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.434994    2580 generic.go:356] "Generic (PLEG): container finished" podID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerID="17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef" exitCode=0
Sep 10 23:15:07.435133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.435075    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"
Sep 10 23:15:07.435133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.435085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" event={"ID":"6cd97e3b-f4e8-49b0-8cae-8485019f582a","Type":"ContainerDied","Data":"17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef"}
Sep 10 23:15:07.435133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.435121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn" event={"ID":"6cd97e3b-f4e8-49b0-8cae-8485019f582a","Type":"ContainerDied","Data":"8a85c89308400c48e9970a0a37dc9881f396020e9ec53e2d07ca61f1dbff99f6"}
Sep 10 23:15:07.435304 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.435150    2580 scope.go:117] "RemoveContainer" containerID="17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef"
Sep 10 23:15:07.449608 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.449583    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb"
Sep 10 23:15:07.449722 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.449629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb" event={"ID":"e5e31206-e26b-4e21-b80c-1dda84dfd7dd","Type":"ContainerDied","Data":"f9aaee3b0b0dd121f5a1d2faf35c1e084bd2a0364407c851823d4fc033db005a"}
Sep 10 23:15:07.449722 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.449667    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9aaee3b0b0dd121f5a1d2faf35c1e084bd2a0364407c851823d4fc033db005a"
Sep 10 23:15:07.458275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.458252    2580 scope.go:117] "RemoveContainer" containerID="dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0"
Sep 10 23:15:07.464073 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.464052    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"]
Sep 10 23:15:07.467573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.467376    2580 scope.go:117] "RemoveContainer" containerID="17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef"
Sep 10 23:15:07.467770 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:15:07.467750    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef\": container with ID starting with 17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef not found: ID does not exist" containerID="17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef"
Sep 10 23:15:07.467868 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.467780    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef"} err="failed to get container status \"17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef\": rpc error: code = NotFound desc = could not find container \"17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef\": container with ID starting with 17db6b8d78fdf10bda28f1130f478a11a2c6a7d4d13f78b91792153d347232ef not found: ID does not exist"
Sep 10 23:15:07.467868 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.467799    2580 scope.go:117] "RemoveContainer" containerID="dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0"
Sep 10 23:15:07.468169 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:15:07.468074    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0\": container with ID starting with dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0 not found: ID does not exist" containerID="dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0"
Sep 10 23:15:07.468169 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.468106    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0"} err="failed to get container status \"dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0\": rpc error: code = NotFound desc = could not find container \"dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0\": container with ID starting with dbcca15c5e990ec6658f83f19f42f19b572a44ce0c434d5137795683f38533c0 not found: ID does not exist"
Sep 10 23:15:07.471613 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.471594    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f897c8bb-openshift-default-657bcd5bf8-bp4gn"]
Sep 10 23:15:07.572819 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.572782    2580 patch_prober.go:28] interesting pod/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 23:15:07.572975 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:07.572842    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 23:15:08.150815 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:08.150779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:08.457115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:08.457014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh" event={"ID":"6d5cf26f-0df0-4f53-9f26-a060be81ed46","Type":"ContainerStarted","Data":"4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32"}
Sep 10 23:15:08.457115 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:08.457079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:08.493755 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:08.493694    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh" podStartSLOduration=2.058187375 podStartE2EDuration="2.493675593s" totalImagesPullingTime="435.488218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:08.489682916 +0000 UTC m=+808.191315862" watchObservedRunningTime="2026-09-10 23:15:08.493675593 +0000 UTC m=+808.195308537"
Sep 10 23:15:08.571975 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:08.571937    2580 patch_prober.go:28] interesting pod/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 23:15:08.572164 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:08.572008    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 23:15:08.839346 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:08.839249    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" path="/var/lib/kubelet/pods/6cd97e3b-f4e8-49b0-8cae-8485019f582a/volumes"
Sep 10 23:15:09.104231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:09.104207    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:15:09.572714 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:09.572622    2580 patch_prober.go:28] interesting pod/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 23:15:09.572714 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:09.572677    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 23:15:10.576833 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:10.576802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:15:10.577780 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:10.577759    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:15:11.076474 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:11.076429    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:15:14.469822 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:14.469796    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:20.794452 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:20.794396    2580 patch_prober.go:28] interesting pod/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:40586->10.134.0.58:15021: read: connection reset by peer" start-of-body=
Sep 10 23:15:20.794899 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:20.794478    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:40586->10.134.0.58:15021: read: connection reset by peer"
Sep 10 23:15:21.543361 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:21.543329    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a444014-619e-4d93-90c6-5942f593164d" containerID="484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f" exitCode=0
Sep 10 23:15:21.543591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:21.543413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" event={"ID":"4a444014-619e-4d93-90c6-5942f593164d","Type":"ContainerDied","Data":"484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f"}
Sep 10 23:15:21.543875 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:21.543862    2580 scope.go:117] "RemoveContainer" containerID="484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f"
Sep 10 23:15:21.545061 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:21.545033    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:21.545118 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:21.545102    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:21.545155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:21.545132    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:22.552192 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:22.552148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" event={"ID":"4a444014-619e-4d93-90c6-5942f593164d","Type":"ContainerStarted","Data":"825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34"}
Sep 10 23:15:22.744807 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:22.744770    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:22.745028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:22.744810    2580 patch_prober.go:28] interesting pod/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 23:15:22.745028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:22.744866    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 23:15:23.745589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:23.745527    2580 patch_prober.go:28] interesting pod/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 23:15:23.746158 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:23.745616    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 23:15:24.745014 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:24.744962    2580 patch_prober.go:28] interesting pod/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 23:15:24.745224 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:24.745027    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 23:15:25.744912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:25.744867    2580 patch_prober.go:28] interesting pod/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 23:15:25.745318 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:25.744956    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 23:15:26.745830 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:26.745781    2580 patch_prober.go:28] interesting pod/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 23:15:26.746333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:26.745867    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 23:15:27.745666 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:27.745628    2580 patch_prober.go:28] interesting pod/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 10 23:15:27.745859 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:27.745700    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 10 23:15:28.749109 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:28.749081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:28.749584 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:28.749563    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:28.750211 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:28.750195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:30.974628 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:30.974596    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:15:33.846528 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.846496    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"]
Sep 10 23:15:33.847142 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847121    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e31206-e26b-4e21-b80c-1dda84dfd7dd" containerName="cleanup"
Sep 10 23:15:33.847246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847144    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e31206-e26b-4e21-b80c-1dda84dfd7dd" containerName="cleanup"
Sep 10 23:15:33.847246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847181    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce8a510-0a87-494d-8980-08c331f3736b" containerName="cleanup"
Sep 10 23:15:33.847246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847189    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce8a510-0a87-494d-8980-08c331f3736b" containerName="cleanup"
Sep 10 23:15:33.847246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847205    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy"
Sep 10 23:15:33.847246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847213    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy"
Sep 10 23:15:33.847246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847230    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy"
Sep 10 23:15:33.847246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847237    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy"
Sep 10 23:15:33.847619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847251    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250a7699-a794-4ddc-a08f-676ceea369a5" containerName="cleanup"
Sep 10 23:15:33.847619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847259    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="250a7699-a794-4ddc-a08f-676ceea369a5" containerName="cleanup"
Sep 10 23:15:33.847619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847358    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce8a510-0a87-494d-8980-08c331f3736b" containerName="cleanup"
Sep 10 23:15:33.847619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847371    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="250a7699-a794-4ddc-a08f-676ceea369a5" containerName="cleanup"
Sep 10 23:15:33.847619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847382    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e31206-e26b-4e21-b80c-1dda84dfd7dd" containerName="cleanup"
Sep 10 23:15:33.847619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847393    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy"
Sep 10 23:15:33.847619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.847403    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd97e3b-f4e8-49b0-8cae-8485019f582a" containerName="istio-proxy"
Sep 10 23:15:33.852565 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.852517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:33.855575 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.855532    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce17e\"/\"simulator-w0-6ce17e-kserve-self-signed-certs\""
Sep 10 23:15:33.855833 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.855561    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce17e\"/\"kube-root-ca.crt\""
Sep 10 23:15:33.855833 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.855570    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce17e\"/\"openshift-service-ca.crt\""
Sep 10 23:15:33.856022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.855619    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce17e\"/\"default-dockercfg-h8dqd\""
Sep 10 23:15:33.858836 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.858813    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"]
Sep 10 23:15:33.875703 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.875653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:33.953508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.953475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tmp-dir\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:33.953718 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.953693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-home\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:33.953814 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.953739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-model-cache\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:33.953814 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.953786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tls-certs\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:33.953941 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.953915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frhh7\" (UniqueName: \"kubernetes.io/projected/e40ccec9-5a51-4fbe-b514-e12a1279f85a-kube-api-access-frhh7\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:33.954077 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:33.954061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-dshm\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.054844 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.054808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frhh7\" (UniqueName: \"kubernetes.io/projected/e40ccec9-5a51-4fbe-b514-e12a1279f85a-kube-api-access-frhh7\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.055067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.054911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-dshm\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.055067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.054972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tmp-dir\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.055067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.055036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-home\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.055364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.055068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-model-cache\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.055364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.055114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tls-certs\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.055673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.055578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-home\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.055673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.055622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tmp-dir\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.055673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.055650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-model-cache\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.058102 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.058082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-dshm\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.058394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.058375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tls-certs\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.061821 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.061797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frhh7\" (UniqueName: \"kubernetes.io/projected/e40ccec9-5a51-4fbe-b514-e12a1279f85a-kube-api-access-frhh7\") pod \"simulator-w0-6ce17e-kserve-5859878bd6-7jpdh\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") " pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.163976 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.163883    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:34.441629 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.441516    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"]
Sep 10 23:15:34.444510 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:15:34.444461    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode40ccec9_5a51_4fbe_b514_e12a1279f85a.slice/crio-4215c044e3e4208c561d82c4d6d55e8ee7103bda1a23e50d4076a74c62c10214 WatchSource:0}: Error finding container 4215c044e3e4208c561d82c4d6d55e8ee7103bda1a23e50d4076a74c62c10214: Status 404 returned error can't find the container with id 4215c044e3e4208c561d82c4d6d55e8ee7103bda1a23e50d4076a74c62c10214
Sep 10 23:15:34.626145 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.626102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh" event={"ID":"e40ccec9-5a51-4fbe-b514-e12a1279f85a","Type":"ContainerStarted","Data":"d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a"}
Sep 10 23:15:34.626145 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.626149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh" event={"ID":"e40ccec9-5a51-4fbe-b514-e12a1279f85a","Type":"ContainerStarted","Data":"4215c044e3e4208c561d82c4d6d55e8ee7103bda1a23e50d4076a74c62c10214"}
Sep 10 23:15:34.645989 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:34.645935    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh" podStartSLOduration=1.645919003 podStartE2EDuration="1.645919003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:34.643588633 +0000 UTC m=+834.345221573" watchObservedRunningTime="2026-09-10 23:15:34.645919003 +0000 UTC m=+834.347551946"
Sep 10 23:15:37.366439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.366407    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"]
Sep 10 23:15:37.366921 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.366716    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh" podUID="6d5cf26f-0df0-4f53-9f26-a060be81ed46" containerName="maas-api" containerID="cri-o://4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32" gracePeriod=30
Sep 10 23:15:37.396500 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.396458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:37.465367 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.465332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:37.534901 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.534791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:37.599621 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.599587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:37.615352 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.615328    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:37.647404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.647330    2580 generic.go:356] "Generic (PLEG): container finished" podID="6d5cf26f-0df0-4f53-9f26-a060be81ed46" containerID="4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32" exitCode=0
Sep 10 23:15:37.647404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.647373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh" event={"ID":"6d5cf26f-0df0-4f53-9f26-a060be81ed46","Type":"ContainerDied","Data":"4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32"}
Sep 10 23:15:37.647404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.647385    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"
Sep 10 23:15:37.647686 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.647411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh" event={"ID":"6d5cf26f-0df0-4f53-9f26-a060be81ed46","Type":"ContainerDied","Data":"f23402f4177464be4f9846bcdb21859edfabf0d8486987e4099f07d8c7508646"}
Sep 10 23:15:37.647686 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.647431    2580 scope.go:117] "RemoveContainer" containerID="4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32"
Sep 10 23:15:37.657846 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.657815    2580 scope.go:117] "RemoveContainer" containerID="4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32"
Sep 10 23:15:37.658157 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:15:37.658138    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32\": container with ID starting with 4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32 not found: ID does not exist" containerID="4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32"
Sep 10 23:15:37.658247 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.658167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32"} err="failed to get container status \"4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32\": rpc error: code = NotFound desc = could not find container \"4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32\": container with ID starting with 4caedde6288aaff619d4cbea8e819bc3ad414d668f398e863d5bd8c4d5a90a32 not found: ID does not exist"
Sep 10 23:15:37.690175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.690148    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7m79\" (UniqueName: \"kubernetes.io/projected/6d5cf26f-0df0-4f53-9f26-a060be81ed46-kube-api-access-j7m79\") pod \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") "
Sep 10 23:15:37.690326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.690216    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-metrics-certs\") pod \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") "
Sep 10 23:15:37.690326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.690291    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-maas-api-tls\") pod \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") "
Sep 10 23:15:37.690522 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.690486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5cf26f-0df0-4f53-9f26-a060be81ed46-openshift-service-ca\") pod \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\" (UID: \"6d5cf26f-0df0-4f53-9f26-a060be81ed46\") "
Sep 10 23:15:37.691103 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.691072    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5cf26f-0df0-4f53-9f26-a060be81ed46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d5cf26f-0df0-4f53-9f26-a060be81ed46" (UID: "6d5cf26f-0df0-4f53-9f26-a060be81ed46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:15:37.694232 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.694183    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d5cf26f-0df0-4f53-9f26-a060be81ed46" (UID: "6d5cf26f-0df0-4f53-9f26-a060be81ed46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:15:37.694337 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.694291    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d5cf26f-0df0-4f53-9f26-a060be81ed46" (UID: "6d5cf26f-0df0-4f53-9f26-a060be81ed46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:15:37.694386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.694328    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5cf26f-0df0-4f53-9f26-a060be81ed46-kube-api-access-j7m79" (OuterVolumeSpecName: "kube-api-access-j7m79") pod "6d5cf26f-0df0-4f53-9f26-a060be81ed46" (UID: "6d5cf26f-0df0-4f53-9f26-a060be81ed46"). InnerVolumeSpecName "kube-api-access-j7m79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:37.707566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.707526    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"]
Sep 10 23:15:37.707820 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.707798    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy" containerID="cri-o://825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34" gracePeriod=30
Sep 10 23:15:37.722218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.722174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:37.791530 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.791497    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7m79\" (UniqueName: \"kubernetes.io/projected/6d5cf26f-0df0-4f53-9f26-a060be81ed46-kube-api-access-j7m79\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:37.791530 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.791526    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:37.791737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.791536    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d5cf26f-0df0-4f53-9f26-a060be81ed46-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:37.791737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.791569    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5cf26f-0df0-4f53-9f26-a060be81ed46-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:37.972812 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.972779    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"]
Sep 10 23:15:37.975404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:37.975381    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7dbe617a-58dc5574b4-469gh"]
Sep 10 23:15:38.142414 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.142386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:38.406809 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.406777    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"]
Sep 10 23:15:38.407210 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.407197    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d5cf26f-0df0-4f53-9f26-a060be81ed46" containerName="maas-api"
Sep 10 23:15:38.407256 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.407212    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d5cf26f-0df0-4f53-9f26-a060be81ed46" containerName="maas-api"
Sep 10 23:15:38.407313 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.407303    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d5cf26f-0df0-4f53-9f26-a060be81ed46" containerName="maas-api"
Sep 10 23:15:38.410081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.410064    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.412230 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.412209    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c5c3fdb6-openshift-default-dockercfg-6jd4x\""
Sep 10 23:15:38.424191 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.424164    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"]
Sep 10 23:15:38.427384 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.427360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:38.500564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.499063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-token\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.500564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.499118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpgq5\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-kube-api-access-kpgq5\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.500564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.499172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-credential-socket\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.500564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.499240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-data\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.500564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.499273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-certs\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.500564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.499314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-podinfo\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.500564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.499348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e877c2-ca0d-4d0b-86bd-f626df148aca-istiod-ca-cert\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.500564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.499412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-envoy\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.500564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.499456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-socket\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.600834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.600800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-data\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.600834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.600840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-certs\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.600868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-podinfo\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.600985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e877c2-ca0d-4d0b-86bd-f626df148aca-istiod-ca-cert\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-envoy\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-socket\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-token\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpgq5\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-kube-api-access-kpgq5\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-data\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601421 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-certs\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601421 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-credential-socket\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601577 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-socket\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601577 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-credential-socket\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.601804 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.601783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e877c2-ca0d-4d0b-86bd-f626df148aca-istiod-ca-cert\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.603460 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.603429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-envoy\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.603683 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.603667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-podinfo\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.608589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.608561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-token\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.608680 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.608611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpgq5\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-kube-api-access-kpgq5\") pod \"e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") " pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.724258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.724185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:38.739044 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.739007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:38.835564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:38.835505    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d5cf26f-0df0-4f53-9f26-a060be81ed46" path="/var/lib/kubelet/pods/6d5cf26f-0df0-4f53-9f26-a060be81ed46/volumes"
Sep 10 23:15:39.002158 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:39.002134    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"]
Sep 10 23:15:39.002767 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:15:39.002741    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e877c2_ca0d_4d0b_86bd_f626df148aca.slice/crio-fa3fc3717a1e0a614e9bc1e8ea86b7a32e1de856ddc80cfca55e4a035adada19 WatchSource:0}: Error finding container fa3fc3717a1e0a614e9bc1e8ea86b7a32e1de856ddc80cfca55e4a035adada19: Status 404 returned error can't find the container with id fa3fc3717a1e0a614e9bc1e8ea86b7a32e1de856ddc80cfca55e4a035adada19
Sep 10 23:15:39.005646 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:39.005613    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:39.005726 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:39.005677    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:39.005726 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:39.005719    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:39.661604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:39.661569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" event={"ID":"16e877c2-ca0d-4d0b-86bd-f626df148aca","Type":"ContainerStarted","Data":"0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336"}
Sep 10 23:15:39.661968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:39.661624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" event={"ID":"16e877c2-ca0d-4d0b-86bd-f626df148aca","Type":"ContainerStarted","Data":"fa3fc3717a1e0a614e9bc1e8ea86b7a32e1de856ddc80cfca55e4a035adada19"}
Sep 10 23:15:39.682296 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:39.682251    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" podStartSLOduration=1.682236639 podStartE2EDuration="1.682236639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:39.679133485 +0000 UTC m=+839.380766407" watchObservedRunningTime="2026-09-10 23:15:39.682236639 +0000 UTC m=+839.383869581"
Sep 10 23:15:39.724967 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:39.724946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:40.731191 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:40.731159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:41.012779 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:41.012701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:41.106741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:41.106707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:41.674299 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:41.674271    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:41.675346 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:41.675329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:42.255769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.255739    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"]
Sep 10 23:15:42.260041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.260018    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.262148 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.262126    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c5c3fdb6-serving-cert\""
Sep 10 23:15:42.262874 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.262843    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:15:42.262986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.262887    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:15:42.262986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.262939    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:15:42.263135 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.263079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5kfc5\""
Sep 10 23:15:42.270047 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.270027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"]
Sep 10 23:15:42.284242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.284209    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:42.338131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.338106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:42.339497 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.339475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qm2b\" (UniqueName: \"kubernetes.io/projected/ed867f12-e122-42ad-8ba2-21705d1ee0a4-kube-api-access-5qm2b\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.339594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.339514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-metrics-certs\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.339720 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.339702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-maas-api-tls\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.339802 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.339787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed867f12-e122-42ad-8ba2-21705d1ee0a4-openshift-service-ca\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.441219 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.441187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qm2b\" (UniqueName: \"kubernetes.io/projected/ed867f12-e122-42ad-8ba2-21705d1ee0a4-kube-api-access-5qm2b\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.441219 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.441224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-metrics-certs\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.441493 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.441421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-maas-api-tls\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.441575 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.441498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed867f12-e122-42ad-8ba2-21705d1ee0a4-openshift-service-ca\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.441633 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:15:42.441592    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-serving-cert: secret "maas-api-e2e-ait-c5c3fdb6-serving-cert" not found
Sep 10 23:15:42.441668 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:15:42.441656    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-maas-api-tls podName:ed867f12-e122-42ad-8ba2-21705d1ee0a4 nodeName:}" failed. No retries permitted until 2026-09-10 23:15:42.941639491 +0000 UTC m=+842.643272412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-maas-api-tls") pod "maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk" (UID: "ed867f12-e122-42ad-8ba2-21705d1ee0a4") : secret "maas-api-e2e-ait-c5c3fdb6-serving-cert" not found
Sep 10 23:15:42.442226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.442205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed867f12-e122-42ad-8ba2-21705d1ee0a4-openshift-service-ca\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.443794 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.443778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-metrics-certs\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.449792 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.449773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qm2b\" (UniqueName: \"kubernetes.io/projected/ed867f12-e122-42ad-8ba2-21705d1ee0a4-kube-api-access-5qm2b\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.719398 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.719363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:42.838254 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.838216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:42.938838 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.938809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:42.946245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.946219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-maas-api-tls\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.949081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.949060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-maas-api-tls\") pod \"maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:42.970995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:42.970934    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:43.047048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a444014-619e-4d93-90c6-5942f593164d-istio-podinfo\") pod \"4a444014-619e-4d93-90c6-5942f593164d\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") "
Sep 10 23:15:43.047048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-credential-socket\") pod \"4a444014-619e-4d93-90c6-5942f593164d\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") "
Sep 10 23:15:43.047272 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047093    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-data\") pod \"4a444014-619e-4d93-90c6-5942f593164d\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") "
Sep 10 23:15:43.047272 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-socket\") pod \"4a444014-619e-4d93-90c6-5942f593164d\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") "
Sep 10 23:15:43.047272 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-certs\") pod \"4a444014-619e-4d93-90c6-5942f593164d\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") "
Sep 10 23:15:43.047420 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-envoy\") pod \"4a444014-619e-4d93-90c6-5942f593164d\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") "
Sep 10 23:15:43.047420 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr74q\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-kube-api-access-qr74q\") pod \"4a444014-619e-4d93-90c6-5942f593164d\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") "
Sep 10 23:15:43.047420 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047384    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a444014-619e-4d93-90c6-5942f593164d" (UID: "4a444014-619e-4d93-90c6-5942f593164d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:43.047579 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-istio-token\") pod \"4a444014-619e-4d93-90c6-5942f593164d\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") "
Sep 10 23:15:43.047579 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a444014-619e-4d93-90c6-5942f593164d-istiod-ca-cert\") pod \"4a444014-619e-4d93-90c6-5942f593164d\" (UID: \"4a444014-619e-4d93-90c6-5942f593164d\") "
Sep 10 23:15:43.047579 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047459    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a444014-619e-4d93-90c6-5942f593164d" (UID: "4a444014-619e-4d93-90c6-5942f593164d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:43.047745 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a444014-619e-4d93-90c6-5942f593164d" (UID: "4a444014-619e-4d93-90c6-5942f593164d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:43.047745 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a444014-619e-4d93-90c6-5942f593164d" (UID: "4a444014-619e-4d93-90c6-5942f593164d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:43.047851 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047816    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a444014-619e-4d93-90c6-5942f593164d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a444014-619e-4d93-90c6-5942f593164d" (UID: "4a444014-619e-4d93-90c6-5942f593164d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:15:43.047934 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047917    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a444014-619e-4d93-90c6-5942f593164d-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:43.047934 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047934    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:43.048054 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047943    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:43.048054 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047952    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:43.048054 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.047961    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:43.049470 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.049439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a444014-619e-4d93-90c6-5942f593164d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a444014-619e-4d93-90c6-5942f593164d" (UID: "4a444014-619e-4d93-90c6-5942f593164d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:15:43.049629 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.049471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a444014-619e-4d93-90c6-5942f593164d" (UID: "4a444014-619e-4d93-90c6-5942f593164d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:43.049721 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.049701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-kube-api-access-qr74q" (OuterVolumeSpecName: "kube-api-access-qr74q") pod "4a444014-619e-4d93-90c6-5942f593164d" (UID: "4a444014-619e-4d93-90c6-5942f593164d"). InnerVolumeSpecName "kube-api-access-qr74q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:43.049777 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.049759    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a444014-619e-4d93-90c6-5942f593164d" (UID: "4a444014-619e-4d93-90c6-5942f593164d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:15:43.148680 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.148645    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr74q\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-kube-api-access-qr74q\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:43.148680 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.148676    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a444014-619e-4d93-90c6-5942f593164d-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:43.148680 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.148686    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a444014-619e-4d93-90c6-5942f593164d-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:43.148906 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.148693    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a444014-619e-4d93-90c6-5942f593164d-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:15:43.175629 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.175600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:43.448836 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.448808    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"]
Sep 10 23:15:43.449824 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:15:43.449801    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded867f12_e122_42ad_8ba2_21705d1ee0a4.slice/crio-e37824fa4c635c78ad9cc9320ae70ae024c30b13e959e8d0ccc7fe9371701331 WatchSource:0}: Error finding container e37824fa4c635c78ad9cc9320ae70ae024c30b13e959e8d0ccc7fe9371701331: Status 404 returned error can't find the container with id e37824fa4c635c78ad9cc9320ae70ae024c30b13e959e8d0ccc7fe9371701331
Sep 10 23:15:43.686414 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.686383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk" event={"ID":"ed867f12-e122-42ad-8ba2-21705d1ee0a4","Type":"ContainerStarted","Data":"e37824fa4c635c78ad9cc9320ae70ae024c30b13e959e8d0ccc7fe9371701331"}
Sep 10 23:15:43.688129 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.688108    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a444014-619e-4d93-90c6-5942f593164d" containerID="825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34" exitCode=0
Sep 10 23:15:43.688226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.688156    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"
Sep 10 23:15:43.688226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.688184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" event={"ID":"4a444014-619e-4d93-90c6-5942f593164d","Type":"ContainerDied","Data":"825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34"}
Sep 10 23:15:43.688226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.688225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j" event={"ID":"4a444014-619e-4d93-90c6-5942f593164d","Type":"ContainerDied","Data":"c07ec9774309fab39d89329f68172808bb1ec6edc373f592390794b4b31f891a"}
Sep 10 23:15:43.688336 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.688247    2580 scope.go:117] "RemoveContainer" containerID="825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34"
Sep 10 23:15:43.698268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.698250    2580 scope.go:117] "RemoveContainer" containerID="484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f"
Sep 10 23:15:43.713255 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.713225    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"]
Sep 10 23:15:43.715005 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.714983    2580 scope.go:117] "RemoveContainer" containerID="825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34"
Sep 10 23:15:43.715300 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:15:43.715282    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34\": container with ID starting with 825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34 not found: ID does not exist" containerID="825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34"
Sep 10 23:15:43.715372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.715306    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34"} err="failed to get container status \"825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34\": rpc error: code = NotFound desc = could not find container \"825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34\": container with ID starting with 825aecfb9eddf5dd64b3fdb7b668767e59723598b7f2e10fa3a4a29a6a432d34 not found: ID does not exist"
Sep 10 23:15:43.715372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.715321    2580 scope.go:117] "RemoveContainer" containerID="484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f"
Sep 10 23:15:43.715587 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:15:43.715564    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f\": container with ID starting with 484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f not found: ID does not exist" containerID="484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f"
Sep 10 23:15:43.715669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.715592    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f"} err="failed to get container status \"484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f\": rpc error: code = NotFound desc = could not find container \"484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f\": container with ID starting with 484f1673460125e5c196e7be47b78d5e612a035b575c35f6070fb1c79bebda3f not found: ID does not exist"
Sep 10 23:15:43.715929 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:43.715910    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7dbe617a-gw-openshift-default-7c5b6dcd7b-8w77j"]
Sep 10 23:15:44.164178 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:44.164150    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:44.164344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:44.164197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:44.176193 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:44.176170    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:44.696022 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:44.695992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk" event={"ID":"ed867f12-e122-42ad-8ba2-21705d1ee0a4","Type":"ContainerStarted","Data":"5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8"}
Sep 10 23:15:44.696470 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:44.696329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:44.706823 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:44.706803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:15:44.712052 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:44.712010    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk" podStartSLOduration=2.248809183 podStartE2EDuration="2.711997908s" totalImagesPullingTime="463.188725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:44.711668518 +0000 UTC m=+844.413301462" watchObservedRunningTime="2026-09-10 23:15:44.711997908 +0000 UTC m=+844.413630856"
Sep 10 23:15:44.834674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:44.834640    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a444014-619e-4d93-90c6-5942f593164d" path="/var/lib/kubelet/pods/4a444014-619e-4d93-90c6-5942f593164d/volumes"
Sep 10 23:15:48.278438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:48.278410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:15:50.708089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:50.708063    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:15:50.996120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:50.995990    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:57.373786 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:57.373700    2580 patch_prober.go:28] interesting pod/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:45204->10.134.0.61:15021: read: connection reset by peer" start-of-body=
Sep 10 23:15:57.374159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:57.373786    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:45204->10.134.0.61:15021: read: connection reset by peer"
Sep 10 23:15:57.786397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:57.786311    2580 generic.go:356] "Generic (PLEG): container finished" podID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerID="0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336" exitCode=0
Sep 10 23:15:57.786397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:57.786387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" event={"ID":"16e877c2-ca0d-4d0b-86bd-f626df148aca","Type":"ContainerDied","Data":"0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336"}
Sep 10 23:15:57.786856 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:57.786841    2580 scope.go:117] "RemoveContainer" containerID="0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336"
Sep 10 23:15:57.787964 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:57.787934    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:57.788021 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:57.788002    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:57.788060 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:57.788034    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:15:58.794802 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:58.794770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" event={"ID":"16e877c2-ca0d-4d0b-86bd-f626df148aca","Type":"ContainerStarted","Data":"52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac"}
Sep 10 23:15:59.725481 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:59.725450    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:15:59.725835 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:59.725814    2580 patch_prober.go:28] interesting pod/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 23:15:59.725980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:15:59.725959    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 23:16:00.725413 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:00.725376    2580 patch_prober.go:28] interesting pod/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 23:16:00.726027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:00.725997    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 23:16:01.724925 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:01.724883    2580 patch_prober.go:28] interesting pod/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 23:16:01.725238 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:01.725205    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 23:16:02.380240 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:02.380207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:02.724768 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:02.724673    2580 patch_prober.go:28] interesting pod/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 23:16:02.725029 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:02.725001    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 23:16:03.171079 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:03.171049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:03.728225 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:03.728198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:16:03.728727 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:03.728589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:16:03.729183 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:03.729165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:16:03.977347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:03.977317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:11.072364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:11.072327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:16:12.848189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:12.848159    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"]
Sep 10 23:16:12.848965 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:12.848936    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk" podUID="ed867f12-e122-42ad-8ba2-21705d1ee0a4" containerName="maas-api" containerID="cri-o://5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8" gracePeriod=30
Sep 10 23:16:12.865800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:12.865771    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:12.939347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:12.939319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:13.002985 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.002953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:13.064076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.064049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:13.092779 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.092759    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:16:13.119146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.119067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:13.169247 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.169212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qm2b\" (UniqueName: \"kubernetes.io/projected/ed867f12-e122-42ad-8ba2-21705d1ee0a4-kube-api-access-5qm2b\") pod \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") "
Sep 10 23:16:13.169428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.169308    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed867f12-e122-42ad-8ba2-21705d1ee0a4-openshift-service-ca\") pod \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") "
Sep 10 23:16:13.169497 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.169436    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-metrics-certs\") pod \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") "
Sep 10 23:16:13.169578 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.169496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-maas-api-tls\") pod \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\" (UID: \"ed867f12-e122-42ad-8ba2-21705d1ee0a4\") "
Sep 10 23:16:13.169720 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.169696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed867f12-e122-42ad-8ba2-21705d1ee0a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed867f12-e122-42ad-8ba2-21705d1ee0a4" (UID: "ed867f12-e122-42ad-8ba2-21705d1ee0a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:13.169870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.169849    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed867f12-e122-42ad-8ba2-21705d1ee0a4-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:13.171594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.171563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed867f12-e122-42ad-8ba2-21705d1ee0a4-kube-api-access-5qm2b" (OuterVolumeSpecName: "kube-api-access-5qm2b") pod "ed867f12-e122-42ad-8ba2-21705d1ee0a4" (UID: "ed867f12-e122-42ad-8ba2-21705d1ee0a4"). InnerVolumeSpecName "kube-api-access-5qm2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:13.171594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.171574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ed867f12-e122-42ad-8ba2-21705d1ee0a4" (UID: "ed867f12-e122-42ad-8ba2-21705d1ee0a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:16:13.171717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.171657    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed867f12-e122-42ad-8ba2-21705d1ee0a4" (UID: "ed867f12-e122-42ad-8ba2-21705d1ee0a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:16:13.270939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.270903    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qm2b\" (UniqueName: \"kubernetes.io/projected/ed867f12-e122-42ad-8ba2-21705d1ee0a4-kube-api-access-5qm2b\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:13.270939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.270932    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:13.270939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.270941    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed867f12-e122-42ad-8ba2-21705d1ee0a4-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:13.437854 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.437756    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"]
Sep 10 23:16:13.438259 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.438238    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy"
Sep 10 23:16:13.438358 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.438262    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy"
Sep 10 23:16:13.438358 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.438301    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed867f12-e122-42ad-8ba2-21705d1ee0a4" containerName="maas-api"
Sep 10 23:16:13.438358 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.438310    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed867f12-e122-42ad-8ba2-21705d1ee0a4" containerName="maas-api"
Sep 10 23:16:13.438517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.438432    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy"
Sep 10 23:16:13.438517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.438447    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed867f12-e122-42ad-8ba2-21705d1ee0a4" containerName="maas-api"
Sep 10 23:16:13.440892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.440871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.443285 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.443252    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-679c94\"/\"openshift-service-ca.crt\""
Sep 10 23:16:13.443285 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.443272    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-679c94\"/\"test-model-679c94-kserve-self-signed-certs\""
Sep 10 23:16:13.443469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.443298    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-679c94\"/\"kube-root-ca.crt\""
Sep 10 23:16:13.443469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.443355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-679c94\"/\"default-dockercfg-pvlcj\""
Sep 10 23:16:13.450227 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.450203    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"]
Sep 10 23:16:13.574599 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.574568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrgrh\" (UniqueName: \"kubernetes.io/projected/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-kube-api-access-vrgrh\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.574780 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.574671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-model-cache\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.574780 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.574701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tmp-dir\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.574780 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.574726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-home\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.574780 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.574749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-dshm\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.574923 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.574863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tls-certs\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.676412 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.676378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tls-certs\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.676642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.676439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrgrh\" (UniqueName: \"kubernetes.io/projected/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-kube-api-access-vrgrh\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.676642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.676496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-model-cache\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.676642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.676511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tmp-dir\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.676642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.676528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-home\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.676642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.676577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-dshm\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.677057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.677023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-home\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.677354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.677061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-model-cache\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.677354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.677097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tmp-dir\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.678924 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.678905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-dshm\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.679068 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.679054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tls-certs\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.684196 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.684175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrgrh\" (UniqueName: \"kubernetes.io/projected/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-kube-api-access-vrgrh\") pod \"test-model-679c94-kserve-648c786fb9-64gsc\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") " pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.746461 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.746379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:13.752396 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.752378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:13.885808 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.885777    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed867f12-e122-42ad-8ba2-21705d1ee0a4" containerID="5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8" exitCode=0
Sep 10 23:16:13.886283 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.885836    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"
Sep 10 23:16:13.886283 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.885864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk" event={"ID":"ed867f12-e122-42ad-8ba2-21705d1ee0a4","Type":"ContainerDied","Data":"5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8"}
Sep 10 23:16:13.886283 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.885916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk" event={"ID":"ed867f12-e122-42ad-8ba2-21705d1ee0a4","Type":"ContainerDied","Data":"e37824fa4c635c78ad9cc9320ae70ae024c30b13e959e8d0ccc7fe9371701331"}
Sep 10 23:16:13.886283 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.885938    2580 scope.go:117] "RemoveContainer" containerID="5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8"
Sep 10 23:16:13.896204 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.896189    2580 scope.go:117] "RemoveContainer" containerID="5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8"
Sep 10 23:16:13.896483 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:16:13.896465    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8\": container with ID starting with 5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8 not found: ID does not exist" containerID="5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8"
Sep 10 23:16:13.896533 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.896497    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8"} err="failed to get container status \"5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8\": rpc error: code = NotFound desc = could not find container \"5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8\": container with ID starting with 5be102f9f915bfeffa8b8b099c78a740db80b0239898c2810d22d26c0b1f47d8 not found: ID does not exist"
Sep 10 23:16:13.907319 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.907285    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"]
Sep 10 23:16:13.911230 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:13.911208    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5c3fdb6-65c6997bdd-dlsmk"]
Sep 10 23:16:14.031057 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.031032    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"]
Sep 10 23:16:14.032853 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:16:14.032824    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod813110f8_3d6f_49ba_a1f3_169ae1fe69d8.slice/crio-b77b200a0f6b91c5fbfc9b2e2e636f4dace18a79430e2ae47eabc395c06d4faf WatchSource:0}: Error finding container b77b200a0f6b91c5fbfc9b2e2e636f4dace18a79430e2ae47eabc395c06d4faf: Status 404 returned error can't find the container with id b77b200a0f6b91c5fbfc9b2e2e636f4dace18a79430e2ae47eabc395c06d4faf
Sep 10 23:16:14.148399 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.148369    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"]
Sep 10 23:16:14.148653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.148630    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy" containerID="cri-o://52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac" gracePeriod=30
Sep 10 23:16:14.178098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.178072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:14.594325 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.594291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:14.834961 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.834927    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed867f12-e122-42ad-8ba2-21705d1ee0a4" path="/var/lib/kubelet/pods/ed867f12-e122-42ad-8ba2-21705d1ee0a4/volumes"
Sep 10 23:16:14.867624 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.867502    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"]
Sep 10 23:16:14.868049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.868029    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy"
Sep 10 23:16:14.868146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.868052    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy"
Sep 10 23:16:14.868203 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.868147    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a444014-619e-4d93-90c6-5942f593164d" containerName="istio-proxy"
Sep 10 23:16:14.870379 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.870357    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:14.872694 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.872674    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-51383f0b-openshift-default-dockercfg-hgdrx\""
Sep 10 23:16:14.885466 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.885443    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"]
Sep 10 23:16:14.893040 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.893014    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:14.896002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.895977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc" event={"ID":"813110f8-3d6f-49ba-a1f3-169ae1fe69d8","Type":"ContainerStarted","Data":"94ab949f248402ee693377634f70c20279d860aa92f96bafdcb090deef23cad9"}
Sep 10 23:16:14.896144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.896017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc" event={"ID":"813110f8-3d6f-49ba-a1f3-169ae1fe69d8","Type":"ContainerStarted","Data":"b77b200a0f6b91c5fbfc9b2e2e636f4dace18a79430e2ae47eabc395c06d4faf"}
Sep 10 23:16:14.916035 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.915970    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc" podStartSLOduration=1.915948696 podStartE2EDuration="1.915948696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:16:14.915851854 +0000 UTC m=+874.617484795" watchObservedRunningTime="2026-09-10 23:16:14.915948696 +0000 UTC m=+874.617581639"
Sep 10 23:16:14.990592 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.990513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istiod-ca-cert\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:14.990776 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.990609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8twk\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-kube-api-access-c8twk\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:14.990776 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.990666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-socket\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:14.990776 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.990739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-podinfo\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:14.990776 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.990767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-data\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:14.990987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.990817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-certs\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:14.990987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.990845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-credential-socket\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:14.990987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.990875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-token\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:14.991146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:14.991030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-envoy\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.091931 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.091890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-podinfo\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.091944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-data\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-certs\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-credential-socket\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-token\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092324 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-envoy\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092324 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istiod-ca-cert\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092324 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8twk\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-kube-api-access-c8twk\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-socket\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-certs\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092955 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istiod-ca-cert\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.092955 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.092884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-credential-socket\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.093297 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.093278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-data\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.093405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.093316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-socket\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.095167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.095147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-podinfo\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.095386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.095369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-envoy\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.100798 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.100774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-token\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.100955 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.100936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8twk\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-kube-api-access-c8twk\") pod \"e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") " pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.181299 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.181214    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:15.340783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.340742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:15.462311 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.462287    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"]
Sep 10 23:16:15.468959 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.468914    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:16:15.469039 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.469002    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:16:15.469087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.469041    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:16:15.905521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.905449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" event={"ID":"4e0f638b-81b4-4f86-87a4-0bc69897ab21","Type":"ContainerStarted","Data":"9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4"}
Sep 10 23:16:15.905962 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.905528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" event={"ID":"4e0f638b-81b4-4f86-87a4-0bc69897ab21","Type":"ContainerStarted","Data":"1e08e912efff7bfaed6723f1ca215bee8d95611fc40d02c5df8726ead214d900"}
Sep 10 23:16:15.925245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:15.925190    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" podStartSLOduration=1.9251747190000001 podStartE2EDuration="1.925174719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:16:15.922095644 +0000 UTC m=+875.623728589" watchObservedRunningTime="2026-09-10 23:16:15.925174719 +0000 UTC m=+875.626807662"
Sep 10 23:16:16.182100 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:16.182010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:16.187382 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:16.187356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:16.911913 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:16.911881    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:16.913018 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:16.912992    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:17.248211 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:17.248133    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:17.349405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:17.349372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:18.439705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.439671    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"]
Sep 10 23:16:18.442751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.442729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.444994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.444975    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-51383f0b-serving-cert\""
Sep 10 23:16:18.445740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.445707    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5kfc5\""
Sep 10 23:16:18.445740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.445735    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:16:18.445924 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.445709    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:16:18.445924 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.445707    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:16:18.455762 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.455691    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"]
Sep 10 23:16:18.476209 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.476175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:18.530796 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.530767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15284897-f42f-440e-8b11-f4f7507293c3-openshift-service-ca\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.530943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.530806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682rn\" (UniqueName: \"kubernetes.io/projected/15284897-f42f-440e-8b11-f4f7507293c3-kube-api-access-682rn\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.530943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.530839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-metrics-certs\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.531011 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.530943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-maas-api-tls\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.576740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.576709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:18.632214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.632179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-maas-api-tls\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.632379 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.632314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15284897-f42f-440e-8b11-f4f7507293c3-openshift-service-ca\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.632379 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:16:18.632322    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-serving-cert: secret "maas-api-e2e-derive-51383f0b-serving-cert" not found
Sep 10 23:16:18.632379 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.632343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682rn\" (UniqueName: \"kubernetes.io/projected/15284897-f42f-440e-8b11-f4f7507293c3-kube-api-access-682rn\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.632379 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.632372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-metrics-certs\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.632627 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:16:18.632402    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-maas-api-tls podName:15284897-f42f-440e-8b11-f4f7507293c3 nodeName:}" failed. No retries permitted until 2026-09-10 23:16:19.132377664 +0000 UTC m=+878.834010588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-maas-api-tls") pod "maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp" (UID: "15284897-f42f-440e-8b11-f4f7507293c3") : secret "maas-api-e2e-derive-51383f0b-serving-cert" not found
Sep 10 23:16:18.633175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.633109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15284897-f42f-440e-8b11-f4f7507293c3-openshift-service-ca\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.634965 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.634940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-metrics-certs\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.641065 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.641040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-682rn\" (UniqueName: \"kubernetes.io/projected/15284897-f42f-440e-8b11-f4f7507293c3-kube-api-access-682rn\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:18.696370 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.696280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:18.731426 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.731393    2580 patch_prober.go:28] interesting pod/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 10 23:16:18.731611 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:18.731454    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:16:19.038008 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.037926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:19.139681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.139648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-maas-api-tls\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:19.140371 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.140346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:19.142210 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.142192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-maas-api-tls\") pod \"maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:19.357238 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.357208    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:19.437004 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.436980    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:16:19.545689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545593    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-credential-socket\") pod \"16e877c2-ca0d-4d0b-86bd-f626df148aca\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") "
Sep 10 23:16:19.545689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-socket\") pod \"16e877c2-ca0d-4d0b-86bd-f626df148aca\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") "
Sep 10 23:16:19.545689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpgq5\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-kube-api-access-kpgq5\") pod \"16e877c2-ca0d-4d0b-86bd-f626df148aca\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") "
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-podinfo\") pod \"16e877c2-ca0d-4d0b-86bd-f626df148aca\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") "
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-envoy\") pod \"16e877c2-ca0d-4d0b-86bd-f626df148aca\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") "
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545780    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-certs\") pod \"16e877c2-ca0d-4d0b-86bd-f626df148aca\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") "
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545807    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-token\") pod \"16e877c2-ca0d-4d0b-86bd-f626df148aca\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") "
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-data\") pod \"16e877c2-ca0d-4d0b-86bd-f626df148aca\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") "
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545918    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e877c2-ca0d-4d0b-86bd-f626df148aca-istiod-ca-cert\") pod \"16e877c2-ca0d-4d0b-86bd-f626df148aca\" (UID: \"16e877c2-ca0d-4d0b-86bd-f626df148aca\") "
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.545946    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "16e877c2-ca0d-4d0b-86bd-f626df148aca" (UID: "16e877c2-ca0d-4d0b-86bd-f626df148aca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.546017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "16e877c2-ca0d-4d0b-86bd-f626df148aca" (UID: "16e877c2-ca0d-4d0b-86bd-f626df148aca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.546147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "16e877c2-ca0d-4d0b-86bd-f626df148aca" (UID: "16e877c2-ca0d-4d0b-86bd-f626df148aca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:19.546266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.546191    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-data" (OuterVolumeSpecName: "istio-data") pod "16e877c2-ca0d-4d0b-86bd-f626df148aca" (UID: "16e877c2-ca0d-4d0b-86bd-f626df148aca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:19.546707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.546371    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:19.546707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.546382    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e877c2-ca0d-4d0b-86bd-f626df148aca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "16e877c2-ca0d-4d0b-86bd-f626df148aca" (UID: "16e877c2-ca0d-4d0b-86bd-f626df148aca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:19.546707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.546389    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:19.546707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.546422    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:19.546707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.546437    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:19.548219 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.548192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "16e877c2-ca0d-4d0b-86bd-f626df148aca" (UID: "16e877c2-ca0d-4d0b-86bd-f626df148aca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:19.548324 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.548262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "16e877c2-ca0d-4d0b-86bd-f626df148aca" (UID: "16e877c2-ca0d-4d0b-86bd-f626df148aca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:16:19.548576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.548555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-token" (OuterVolumeSpecName: "istio-token") pod "16e877c2-ca0d-4d0b-86bd-f626df148aca" (UID: "16e877c2-ca0d-4d0b-86bd-f626df148aca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:19.548674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.548583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-kube-api-access-kpgq5" (OuterVolumeSpecName: "kube-api-access-kpgq5") pod "16e877c2-ca0d-4d0b-86bd-f626df148aca" (UID: "16e877c2-ca0d-4d0b-86bd-f626df148aca"). InnerVolumeSpecName "kube-api-access-kpgq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:19.642211 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.642184    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"]
Sep 10 23:16:19.644077 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:16:19.644053    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15284897_f42f_440e_8b11_f4f7507293c3.slice/crio-502aac1552c1b4c2a56a24527f3a228d64165bbb366a4b685dc935847d980959 WatchSource:0}: Error finding container 502aac1552c1b4c2a56a24527f3a228d64165bbb366a4b685dc935847d980959: Status 404 returned error can't find the container with id 502aac1552c1b4c2a56a24527f3a228d64165bbb366a4b685dc935847d980959
Sep 10 23:16:19.647771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.647751    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpgq5\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-kube-api-access-kpgq5\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:19.647870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.647774    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:19.647870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.647784    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:19.647870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.647792    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e877c2-ca0d-4d0b-86bd-f626df148aca-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:19.647870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.647801    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e877c2-ca0d-4d0b-86bd-f626df148aca-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:19.932027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.931991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp" event={"ID":"15284897-f42f-440e-8b11-f4f7507293c3","Type":"ContainerStarted","Data":"502aac1552c1b4c2a56a24527f3a228d64165bbb366a4b685dc935847d980959"}
Sep 10 23:16:19.933811 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.933786    2580 generic.go:356] "Generic (PLEG): container finished" podID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerID="52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac" exitCode=0
Sep 10 23:16:19.933917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.933819    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"
Sep 10 23:16:19.933917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.933862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" event={"ID":"16e877c2-ca0d-4d0b-86bd-f626df148aca","Type":"ContainerDied","Data":"52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac"}
Sep 10 23:16:19.933917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.933904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq" event={"ID":"16e877c2-ca0d-4d0b-86bd-f626df148aca","Type":"ContainerDied","Data":"fa3fc3717a1e0a614e9bc1e8ea86b7a32e1de856ddc80cfca55e4a035adada19"}
Sep 10 23:16:19.934047 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.933927    2580 scope.go:117] "RemoveContainer" containerID="52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac"
Sep 10 23:16:19.944087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.944077    2580 scope.go:117] "RemoveContainer" containerID="0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336"
Sep 10 23:16:19.953239 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.953223    2580 scope.go:117] "RemoveContainer" containerID="52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac"
Sep 10 23:16:19.953485 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:16:19.953470    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac\": container with ID starting with 52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac not found: ID does not exist" containerID="52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac"
Sep 10 23:16:19.953532 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.953492    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac"} err="failed to get container status \"52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac\": rpc error: code = NotFound desc = could not find container \"52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac\": container with ID starting with 52336b80d1537d4b49771e4645a6b2dee74d7ae1314ba6f1f650a598afdd91ac not found: ID does not exist"
Sep 10 23:16:19.953532 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.953506    2580 scope.go:117] "RemoveContainer" containerID="0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336"
Sep 10 23:16:19.953788 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:16:19.953768    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336\": container with ID starting with 0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336 not found: ID does not exist" containerID="0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336"
Sep 10 23:16:19.953836 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.953797    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336"} err="failed to get container status \"0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336\": rpc error: code = NotFound desc = could not find container \"0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336\": container with ID starting with 0612e67af2ba2657ef153ef9c3754e5a99c6803d2eaa7aae501acb93f4255336 not found: ID does not exist"
Sep 10 23:16:19.972153 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.972133    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"]
Sep 10 23:16:19.989617 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:19.989592    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c5c3fdb6-openshift-default-79c6fbd5b8-j9rlq"]
Sep 10 23:16:20.834616 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:20.834585    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" path="/var/lib/kubelet/pods/16e877c2-ca0d-4d0b-86bd-f626df148aca/volumes"
Sep 10 23:16:20.940944 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:20.940905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp" event={"ID":"15284897-f42f-440e-8b11-f4f7507293c3","Type":"ContainerStarted","Data":"a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9"}
Sep 10 23:16:20.941151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:20.940975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:20.959644 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:20.959594    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp" podStartSLOduration=2.5024715730000002 podStartE2EDuration="2.959577278s" totalImagesPullingTime="457.105705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:16:20.95834253 +0000 UTC m=+880.659975512" watchObservedRunningTime="2026-09-10 23:16:20.959577278 +0000 UTC m=+880.661210222"
Sep 10 23:16:21.820366 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:21.820333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:23.752746 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:23.752710    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:23.753173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:23.753039    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:23.765326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:23.765297    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:23.971195 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:23.971164    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:16:26.953399 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:26.953370    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:33.878290 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:33.878208    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:16:37.280219 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:37.280184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:39.424399 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:39.424336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:43.458026 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:43.457986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:45.099551 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:45.099519    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerID="9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4" exitCode=0
Sep 10 23:16:45.099972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:45.099582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" event={"ID":"4e0f638b-81b4-4f86-87a4-0bc69897ab21","Type":"ContainerDied","Data":"9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4"}
Sep 10 23:16:45.100018 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:45.099993    2580 scope.go:117] "RemoveContainer" containerID="9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4"
Sep 10 23:16:45.101253 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:45.101224    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:16:45.101306 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:45.101291    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:16:45.101347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:45.101323    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:16:46.107518 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:46.107480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" event={"ID":"4e0f638b-81b4-4f86-87a4-0bc69897ab21","Type":"ContainerStarted","Data":"c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192"}
Sep 10 23:16:46.181987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:46.181956    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:46.182941 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:46.182919    2580 patch_prober.go:28] interesting pod/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 23:16:46.183034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:46.182969    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 23:16:46.912007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:46.911969    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:47.182443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:47.182357    2580 patch_prober.go:28] interesting pod/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 23:16:47.182443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:47.182423    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 23:16:47.305869 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:47.305837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:47.859679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:47.859650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:47.941762 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:47.941731    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:48.001297 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:48.001260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:48.070197 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:48.070170    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:48.148831 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:48.148728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:48.182520 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:48.182473    2580 patch_prober.go:28] interesting pod/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 23:16:48.183003 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:48.182570    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 23:16:48.418355 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:48.418251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:48.543290 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:48.543250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:49.111953 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.111903    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"]
Sep 10 23:16:49.112150 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.112127    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp" podUID="15284897-f42f-440e-8b11-f4f7507293c3" containerName="maas-api" containerID="cri-o://a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9" gracePeriod=30
Sep 10 23:16:49.147131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.147011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:49.182527 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.182494    2580 patch_prober.go:28] interesting pod/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 23:16:49.182734 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.182582    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 23:16:49.246938 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.246843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:49.341117 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.341030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:49.394887 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.394848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:49.406215 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.406193    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:49.487568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.485569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:49.487568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.486191    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-682rn\" (UniqueName: \"kubernetes.io/projected/15284897-f42f-440e-8b11-f4f7507293c3-kube-api-access-682rn\") pod \"15284897-f42f-440e-8b11-f4f7507293c3\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") "
Sep 10 23:16:49.487568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.486243    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-maas-api-tls\") pod \"15284897-f42f-440e-8b11-f4f7507293c3\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") "
Sep 10 23:16:49.487568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.486313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15284897-f42f-440e-8b11-f4f7507293c3-openshift-service-ca\") pod \"15284897-f42f-440e-8b11-f4f7507293c3\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") "
Sep 10 23:16:49.487568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.486474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-metrics-certs\") pod \"15284897-f42f-440e-8b11-f4f7507293c3\" (UID: \"15284897-f42f-440e-8b11-f4f7507293c3\") "
Sep 10 23:16:49.491589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.488329    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"]
Sep 10 23:16:49.491589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.488558    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy" containerID="cri-o://c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192" gracePeriod=30
Sep 10 23:16:49.491589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.489407    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15284897-f42f-440e-8b11-f4f7507293c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15284897-f42f-440e-8b11-f4f7507293c3" (UID: "15284897-f42f-440e-8b11-f4f7507293c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:49.495049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.495020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15284897-f42f-440e-8b11-f4f7507293c3" (UID: "15284897-f42f-440e-8b11-f4f7507293c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:16:49.496937 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.496907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "15284897-f42f-440e-8b11-f4f7507293c3" (UID: "15284897-f42f-440e-8b11-f4f7507293c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:16:49.497229 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.497201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15284897-f42f-440e-8b11-f4f7507293c3-kube-api-access-682rn" (OuterVolumeSpecName: "kube-api-access-682rn") pod "15284897-f42f-440e-8b11-f4f7507293c3" (UID: "15284897-f42f-440e-8b11-f4f7507293c3"). InnerVolumeSpecName "kube-api-access-682rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:49.587775 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.587744    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:49.587986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.587969    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-682rn\" (UniqueName: \"kubernetes.io/projected/15284897-f42f-440e-8b11-f4f7507293c3-kube-api-access-682rn\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:49.588098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.588082    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15284897-f42f-440e-8b11-f4f7507293c3-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:49.588185 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:49.588175    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15284897-f42f-440e-8b11-f4f7507293c3-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:50.112423 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.112392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:50.167948 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.167918    2580 generic.go:356] "Generic (PLEG): container finished" podID="15284897-f42f-440e-8b11-f4f7507293c3" containerID="a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9" exitCode=0
Sep 10 23:16:50.168135 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.167960    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"
Sep 10 23:16:50.168135 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.168007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp" event={"ID":"15284897-f42f-440e-8b11-f4f7507293c3","Type":"ContainerDied","Data":"a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9"}
Sep 10 23:16:50.168135 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.168063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp" event={"ID":"15284897-f42f-440e-8b11-f4f7507293c3","Type":"ContainerDied","Data":"502aac1552c1b4c2a56a24527f3a228d64165bbb366a4b685dc935847d980959"}
Sep 10 23:16:50.168135 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.168088    2580 scope.go:117] "RemoveContainer" containerID="a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9"
Sep 10 23:16:50.177589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.177571    2580 scope.go:117] "RemoveContainer" containerID="a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9"
Sep 10 23:16:50.177889 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:16:50.177868    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9\": container with ID starting with a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9 not found: ID does not exist" containerID="a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9"
Sep 10 23:16:50.177981 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.177896    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9"} err="failed to get container status \"a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9\": rpc error: code = NotFound desc = could not find container \"a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9\": container with ID starting with a2013c5969597440e3573c25a4708eba3f1c60371a34025ee813ef42d77a4aa9 not found: ID does not exist"
Sep 10 23:16:50.188615 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.188591    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"]
Sep 10 23:16:50.191288 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.191261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:50.194663 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.194640    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51383f0b-657cb6bffc-klrhp"]
Sep 10 23:16:50.253402 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.253370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:50.842737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:50.842701    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15284897-f42f-440e-8b11-f4f7507293c3" path="/var/lib/kubelet/pods/15284897-f42f-440e-8b11-f4f7507293c3/volumes"
Sep 10 23:16:51.617376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:51.617339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:53.488346 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:53.488311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:57.152456 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.152433    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:57.214252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.214219    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerID="c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192" exitCode=0
Sep 10 23:16:57.214450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.214276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"
Sep 10 23:16:57.214450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.214291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" event={"ID":"4e0f638b-81b4-4f86-87a4-0bc69897ab21","Type":"ContainerDied","Data":"c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192"}
Sep 10 23:16:57.214450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.214331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2" event={"ID":"4e0f638b-81b4-4f86-87a4-0bc69897ab21","Type":"ContainerDied","Data":"1e08e912efff7bfaed6723f1ca215bee8d95611fc40d02c5df8726ead214d900"}
Sep 10 23:16:57.214450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.214354    2580 scope.go:117] "RemoveContainer" containerID="c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192"
Sep 10 23:16:57.224312 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.224291    2580 scope.go:117] "RemoveContainer" containerID="9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4"
Sep 10 23:16:57.233553 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.233514    2580 scope.go:117] "RemoveContainer" containerID="c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192"
Sep 10 23:16:57.233831 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:16:57.233809    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192\": container with ID starting with c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192 not found: ID does not exist" containerID="c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192"
Sep 10 23:16:57.233882 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.233840    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192"} err="failed to get container status \"c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192\": rpc error: code = NotFound desc = could not find container \"c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192\": container with ID starting with c8f3020bf5fe6ea4c787458fe66821ba03c20e40dfb28a5f4e0acdddf66d7192 not found: ID does not exist"
Sep 10 23:16:57.233882 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.233857    2580 scope.go:117] "RemoveContainer" containerID="9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4"
Sep 10 23:16:57.234122 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:16:57.234105    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4\": container with ID starting with 9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4 not found: ID does not exist" containerID="9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4"
Sep 10 23:16:57.234177 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.234128    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4"} err="failed to get container status \"9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4\": rpc error: code = NotFound desc = could not find container \"9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4\": container with ID starting with 9ec30990d532c7a0fd1fd9c22c831fb68f00fb883fb9870e01b0e99aadbb76d4 not found: ID does not exist"
Sep 10 23:16:57.274761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.274664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istiod-ca-cert\") pod \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") "
Sep 10 23:16:57.274761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.274714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-credential-socket\") pod \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") "
Sep 10 23:16:57.274761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.274732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-envoy\") pod \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") "
Sep 10 23:16:57.274761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.274765    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-podinfo\") pod \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") "
Sep 10 23:16:57.275138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.274850    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-socket\") pod \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") "
Sep 10 23:16:57.275138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.274887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-certs\") pod \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") "
Sep 10 23:16:57.275138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.274922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-data\") pod \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") "
Sep 10 23:16:57.275138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.274946    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-token\") pod \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") "
Sep 10 23:16:57.275138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.275000    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8twk\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-kube-api-access-c8twk\") pod \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\" (UID: \"4e0f638b-81b4-4f86-87a4-0bc69897ab21\") "
Sep 10 23:16:57.275390 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.274995    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e0f638b-81b4-4f86-87a4-0bc69897ab21" (UID: "4e0f638b-81b4-4f86-87a4-0bc69897ab21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:57.275444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.275205    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e0f638b-81b4-4f86-87a4-0bc69897ab21" (UID: "4e0f638b-81b4-4f86-87a4-0bc69897ab21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:57.275444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.275316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e0f638b-81b4-4f86-87a4-0bc69897ab21" (UID: "4e0f638b-81b4-4f86-87a4-0bc69897ab21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:57.275444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.275371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e0f638b-81b4-4f86-87a4-0bc69897ab21" (UID: "4e0f638b-81b4-4f86-87a4-0bc69897ab21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:57.275629 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.275457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e0f638b-81b4-4f86-87a4-0bc69897ab21" (UID: "4e0f638b-81b4-4f86-87a4-0bc69897ab21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:57.277240 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.277205    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e0f638b-81b4-4f86-87a4-0bc69897ab21" (UID: "4e0f638b-81b4-4f86-87a4-0bc69897ab21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:16:57.277355 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.277307    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e0f638b-81b4-4f86-87a4-0bc69897ab21" (UID: "4e0f638b-81b4-4f86-87a4-0bc69897ab21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:57.277650 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.277622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e0f638b-81b4-4f86-87a4-0bc69897ab21" (UID: "4e0f638b-81b4-4f86-87a4-0bc69897ab21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:57.277747 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.277667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-kube-api-access-c8twk" (OuterVolumeSpecName: "kube-api-access-c8twk") pod "4e0f638b-81b4-4f86-87a4-0bc69897ab21" (UID: "4e0f638b-81b4-4f86-87a4-0bc69897ab21"). InnerVolumeSpecName "kube-api-access-c8twk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:57.376038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.376003    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:57.376038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.376032    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:57.376038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.376043    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:57.376268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.376050    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:57.376268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.376058    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8twk\" (UniqueName: \"kubernetes.io/projected/4e0f638b-81b4-4f86-87a4-0bc69897ab21-kube-api-access-c8twk\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:57.376268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.376067    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:57.376268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.376075    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:57.376268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.376082    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:57.376268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.376089    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0f638b-81b4-4f86-87a4-0bc69897ab21-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:16:57.536759 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.536679    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"]
Sep 10 23:16:57.541009 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.540984    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-51383f0b-openshift-default-54994cc4b4-4szj2"]
Sep 10 23:16:57.678995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.678967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:57.740993 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.740964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:57.814246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.814161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:57.913475 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:57.913436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:58.013913 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:58.013879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:58.331880 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:58.331819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:58.835003 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:16:58.834970    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" path="/var/lib/kubelet/pods/4e0f638b-81b4-4f86-87a4-0bc69897ab21/volumes"
Sep 10 23:17:00.270483 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:00.270449    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:17:24.206950 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:24.206917    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:25.382869 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:25.382831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:30.079641 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:30.079607    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:17:30.674173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:30.674143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:30.775508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:30.775473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:30.874817 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:30.874782    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:31.768588 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:31.768517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:31.997904 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:31.997871    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"]
Sep 10 23:17:31.998117 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:31.998092    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy" containerID="cri-o://2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697" gracePeriod=30
Sep 10 23:17:32.014838 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.014805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:32.441443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.441396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:32.725394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725318    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"]
Sep 10 23:17:32.725736 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725723    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy"
Sep 10 23:17:32.725797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725752    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy"
Sep 10 23:17:32.725797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725762    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy"
Sep 10 23:17:32.725797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725769    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy"
Sep 10 23:17:32.725797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725776    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15284897-f42f-440e-8b11-f4f7507293c3" containerName="maas-api"
Sep 10 23:17:32.725797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725782    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15284897-f42f-440e-8b11-f4f7507293c3" containerName="maas-api"
Sep 10 23:17:32.725797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725790    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy"
Sep 10 23:17:32.725797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725795    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy"
Sep 10 23:17:32.726016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725809    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy"
Sep 10 23:17:32.726016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725814    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy"
Sep 10 23:17:32.726016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725877    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy"
Sep 10 23:17:32.726016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725886    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy"
Sep 10 23:17:32.726016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725891    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0f638b-81b4-4f86-87a4-0bc69897ab21" containerName="istio-proxy"
Sep 10 23:17:32.726016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725897    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15284897-f42f-440e-8b11-f4f7507293c3" containerName="maas-api"
Sep 10 23:17:32.726016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.725905    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e877c2-ca0d-4d0b-86bd-f626df148aca" containerName="istio-proxy"
Sep 10 23:17:32.728599 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.728580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.730879 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.730855    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b3d9be-openshift-default-dockercfg-8k4fb\""
Sep 10 23:17:32.741400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.741373    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"]
Sep 10 23:17:32.748261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.748231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:32.828139 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.828099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-podinfo\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.828708 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.828683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-envoy\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.828910 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.828893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-token\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.829023 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.829010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-certs\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.829127 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.829115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-socket\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.829221 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.829208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpgq\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-kube-api-access-hnpgq\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.829332 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.829318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-data\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.829444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.829431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-credential-socket\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.829567 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.829535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8588db66-e7c9-4060-b143-8d9c05e8315a-istiod-ca-cert\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931095 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-socket\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpgq\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-kube-api-access-hnpgq\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-data\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931468 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-socket\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-credential-socket\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-data\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931768 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8588db66-e7c9-4060-b143-8d9c05e8315a-istiod-ca-cert\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931842 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-podinfo\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-envoy\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.931947 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.931898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-credential-socket\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.932020 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.932003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-token\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.932078 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.932039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-certs\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.932347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.932320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-certs\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.932996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.932972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8588db66-e7c9-4060-b143-8d9c05e8315a-istiod-ca-cert\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.934813 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.934790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-envoy\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.935094 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.935073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-podinfo\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.939427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.939400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-token\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:32.939518 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:32.939502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpgq\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-kube-api-access-hnpgq\") pod \"e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") " pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:33.039904 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:33.039808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:33.198121 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:33.198081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:33.320672 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:33.320648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"]
Sep 10 23:17:33.323107 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:17:33.323074    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8588db66_e7c9_4060_b143_8d9c05e8315a.slice/crio-ec9bd0d8d593a6fc9d1e1a2db94ef73f8d1951212b30ed06eed4e92084e8d44e WatchSource:0}: Error finding container ec9bd0d8d593a6fc9d1e1a2db94ef73f8d1951212b30ed06eed4e92084e8d44e: Status 404 returned error can't find the container with id ec9bd0d8d593a6fc9d1e1a2db94ef73f8d1951212b30ed06eed4e92084e8d44e
Sep 10 23:17:33.325974 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:33.325942    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:17:33.326055 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:33.326009    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:17:33.326055 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:33.326037    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:17:33.437216 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:33.437168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" event={"ID":"8588db66-e7c9-4060-b143-8d9c05e8315a","Type":"ContainerStarted","Data":"77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33"}
Sep 10 23:17:33.437423 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:33.437247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" event={"ID":"8588db66-e7c9-4060-b143-8d9c05e8315a","Type":"ContainerStarted","Data":"ec9bd0d8d593a6fc9d1e1a2db94ef73f8d1951212b30ed06eed4e92084e8d44e"}
Sep 10 23:17:33.458733 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:33.458672    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" podStartSLOduration=1.45864987 podStartE2EDuration="1.45864987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:17:33.455283784 +0000 UTC m=+953.156916729" watchObservedRunningTime="2026-09-10 23:17:33.45864987 +0000 UTC m=+953.160282813"
Sep 10 23:17:34.040698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:34.040662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:34.046067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:34.046041    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:34.444418 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:34.444388    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:34.445470 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:34.445443    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:17:35.332048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:35.332009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:35.428301 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:35.428263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:36.596512 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:36.596479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:36.696284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:36.696241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:36.857479 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:36.857391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:37.169854 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.169814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:37.263353 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.263326    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:17:37.271272 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.271234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:37.381036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.380931    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-podinfo\") pod \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") "
Sep 10 23:17:37.381036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.380990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pszx4\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-kube-api-access-pszx4\") pod \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") "
Sep 10 23:17:37.381036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.381031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-data\") pod \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") "
Sep 10 23:17:37.381330 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.381093    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-token\") pod \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") "
Sep 10 23:17:37.381330 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.381117    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istiod-ca-cert\") pod \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") "
Sep 10 23:17:37.381330 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.381156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-envoy\") pod \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") "
Sep 10 23:17:37.381330 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.381220    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-credential-socket\") pod \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") "
Sep 10 23:17:37.381330 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.381305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-certs\") pod \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") "
Sep 10 23:17:37.381625 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.381337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-socket\") pod \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\" (UID: \"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75\") "
Sep 10 23:17:37.383006 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.382010    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" (UID: "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:17:37.383006 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.382244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" (UID: "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:17:37.383006 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.382268    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" (UID: "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:17:37.383006 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.382431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-data" (OuterVolumeSpecName: "istio-data") pod "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" (UID: "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:17:37.383006 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.382952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" (UID: "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:17:37.407897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.394678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" (UID: "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:17:37.407897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.396943    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-token" (OuterVolumeSpecName: "istio-token") pod "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" (UID: "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:17:37.407897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.407679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-kube-api-access-pszx4" (OuterVolumeSpecName: "kube-api-access-pszx4") pod "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" (UID: "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75"). InnerVolumeSpecName "kube-api-access-pszx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:17:37.411179 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.410725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" (UID: "1bedfa5d-56c4-43eb-95a2-8c83f1e28b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:17:37.463715 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.463689    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"
Sep 10 23:17:37.463715 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.463705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" event={"ID":"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75","Type":"ContainerDied","Data":"2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697"}
Sep 10 23:17:37.463957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.463762    2580 scope.go:117] "RemoveContainer" containerID="2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697"
Sep 10 23:17:37.463957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.463680    2580 generic.go:356] "Generic (PLEG): container finished" podID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerID="2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697" exitCode=0
Sep 10 23:17:37.463957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.463870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9" event={"ID":"1bedfa5d-56c4-43eb-95a2-8c83f1e28b75","Type":"ContainerDied","Data":"ec8186850c62aee9249e4f19d664975acfdf501397df06853a50905964bd8aea"}
Sep 10 23:17:37.474137 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.474115    2580 scope.go:117] "RemoveContainer" containerID="d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3"
Sep 10 23:17:37.482391 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.482360    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:17:37.482391 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.482388    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:17:37.482514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.482397    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:17:37.482514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.482407    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:17:37.482514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.482415    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:17:37.482514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.482423    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pszx4\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-kube-api-access-pszx4\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:17:37.482514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.482432    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:17:37.482514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.482439    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:17:37.482514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.482447    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:17:37.483214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.483196    2580 scope.go:117] "RemoveContainer" containerID="2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697"
Sep 10 23:17:37.483735 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:17:37.483601    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697\": container with ID starting with 2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697 not found: ID does not exist" containerID="2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697"
Sep 10 23:17:37.483735 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.483639    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697"} err="failed to get container status \"2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697\": rpc error: code = NotFound desc = could not find container \"2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697\": container with ID starting with 2ac60d0e9ce634d645cfd6de63bcd09ec400f6bfd20be703861d2c2e31644697 not found: ID does not exist"
Sep 10 23:17:37.483735 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.483661    2580 scope.go:117] "RemoveContainer" containerID="d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3"
Sep 10 23:17:37.484032 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:17:37.484010    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3\": container with ID starting with d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3 not found: ID does not exist" containerID="d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3"
Sep 10 23:17:37.484083 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.484042    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3"} err="failed to get container status \"d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3\": rpc error: code = NotFound desc = could not find container \"d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3\": container with ID starting with d3e84fb3e66c232a92f3e808d32e079e90a579d7336a33150b81c93e7f1587d3 not found: ID does not exist"
Sep 10 23:17:37.486449 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.486430    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"]
Sep 10 23:17:37.492483 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:37.492460    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-689fe3-openshift-default-c89d5bbd4-d8fj9"]
Sep 10 23:17:38.835248 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:38.835217    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" path="/var/lib/kubelet/pods/1bedfa5d-56c4-43eb-95a2-8c83f1e28b75/volumes"
Sep 10 23:17:44.967969 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:44.967935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:46.666791 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:46.666756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:49.742044 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.741999    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"]
Sep 10 23:17:49.742658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.742639    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy"
Sep 10 23:17:49.742712 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.742661    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy"
Sep 10 23:17:49.742712 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.742685    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy"
Sep 10 23:17:49.742712 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.742694    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy"
Sep 10 23:17:49.742842 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.742800    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy"
Sep 10 23:17:49.742842 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.742813    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bedfa5d-56c4-43eb-95a2-8c83f1e28b75" containerName="istio-proxy"
Sep 10 23:17:49.748119 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.748092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:49.750499 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.750465    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-568856\"/\"default-dockercfg-svrlx\""
Sep 10 23:17:49.750704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.750535    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-568856\"/\"openshift-service-ca.crt\""
Sep 10 23:17:49.751276 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.751252    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-568856\"/\"test-model-568856-kserve-self-signed-certs\""
Sep 10 23:17:49.751276 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.751268    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-568856\"/\"kube-root-ca.crt\""
Sep 10 23:17:49.753349 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.753326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"]
Sep 10 23:17:49.767828 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.767777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:49.913698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.913663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-dshm\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:49.913873 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.913760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tmp-dir\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:49.913873 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.913796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-home\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:49.913873 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.913862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tls-certs\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:49.913980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.913921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvln\" (UniqueName: \"kubernetes.io/projected/f1b13246-fbbb-497a-8e6f-c99c94e9093d-kube-api-access-mlvln\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:49.913980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:49.913941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-model-cache\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.015242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.015147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-dshm\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.015242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.015207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tmp-dir\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.015242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.015237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-home\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.015533 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.015267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tls-certs\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.015533 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.015313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvln\" (UniqueName: \"kubernetes.io/projected/f1b13246-fbbb-497a-8e6f-c99c94e9093d-kube-api-access-mlvln\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.015533 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.015331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-model-cache\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.015737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.015698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-home\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.015869 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.015848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tmp-dir\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.015939 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.015919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-model-cache\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.017719 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.017693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-dshm\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.018117 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.018099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tls-certs\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.023307 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.023280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvln\" (UniqueName: \"kubernetes.io/projected/f1b13246-fbbb-497a-8e6f-c99c94e9093d-kube-api-access-mlvln\") pod \"test-model-568856-kserve-5964b7957c-wqc2g\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") " pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.060407 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.060372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:17:50.335633 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.335599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"]
Sep 10 23:17:50.337136 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:17:50.337102    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1b13246_fbbb_497a_8e6f_c99c94e9093d.slice/crio-0f145f292ad58a7322a92da3677f85f7e7fa4a993a32fc590ebf7a91302a38a8 WatchSource:0}: Error finding container 0f145f292ad58a7322a92da3677f85f7e7fa4a993a32fc590ebf7a91302a38a8: Status 404 returned error can't find the container with id 0f145f292ad58a7322a92da3677f85f7e7fa4a993a32fc590ebf7a91302a38a8
Sep 10 23:17:50.477849 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.477810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:17:50.550274 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.550234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g" event={"ID":"f1b13246-fbbb-497a-8e6f-c99c94e9093d","Type":"ContainerStarted","Data":"de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c"}
Sep 10 23:17:50.550695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.550670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g" event={"ID":"f1b13246-fbbb-497a-8e6f-c99c94e9093d","Type":"ContainerStarted","Data":"0f145f292ad58a7322a92da3677f85f7e7fa4a993a32fc590ebf7a91302a38a8"}
Sep 10 23:17:50.570175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:50.570111    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g" podStartSLOduration=1.570087544 podStartE2EDuration="1.570087544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:17:50.567630044 +0000 UTC m=+970.269262988" watchObservedRunningTime="2026-09-10 23:17:50.570087544 +0000 UTC m=+970.271720488"
Sep 10 23:17:52.665359 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:52.665330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:52.723709 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:52.723670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:52.800007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:52.799968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:52.904623 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:17:52.904583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:00.061106 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:00.061071    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:18:00.061622 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:00.061122    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:18:00.073970 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:00.073941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:18:00.259416 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:00.259390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:00.627918 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:00.627881    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:18:01.618165 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:01.618067    2580 generic.go:356] "Generic (PLEG): container finished" podID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerID="77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33" exitCode=0
Sep 10 23:18:01.618709 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:01.618159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" event={"ID":"8588db66-e7c9-4060-b143-8d9c05e8315a","Type":"ContainerDied","Data":"77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33"}
Sep 10 23:18:01.618834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:01.618816    2580 scope.go:117] "RemoveContainer" containerID="77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33"
Sep 10 23:18:01.620144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:01.620110    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:18:01.620241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:01.620198    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:18:01.620301 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:01.620243    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:18:02.630349 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:02.630311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" event={"ID":"8588db66-e7c9-4060-b143-8d9c05e8315a","Type":"ContainerStarted","Data":"28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50"}
Sep 10 23:18:03.040971 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:03.040872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:18:03.041374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:03.041347    2580 patch_prober.go:28] interesting pod/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 23:18:03.041624 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:03.041596    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 23:18:03.340726 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:03.340600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:04.041353 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:04.041304    2580 patch_prober.go:28] interesting pod/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 23:18:04.041832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:04.041379    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 23:18:04.098093 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:04.098058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:04.445458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:04.445431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:18:05.040818 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:05.040779    2580 patch_prober.go:28] interesting pod/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 23:18:05.041119 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:05.041092    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 23:18:06.040976 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:06.040928    2580 patch_prober.go:28] interesting pod/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 23:18:06.041458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:06.041000    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 23:18:07.040277 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:07.040240    2580 patch_prober.go:28] interesting pod/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 10 23:18:07.040614 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:07.040588    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 10 23:18:08.044277 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:08.044244    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:18:08.045328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:08.045312    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:18:10.277961 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:10.277920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:10.380219 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:10.380181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:10.478553 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:10.478518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:10.578923 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:10.578838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:11.184284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:11.184251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:11.283084 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:11.283053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:11.393173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:11.393136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:12.803108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:12.803070    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:13.580853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:13.580819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:13.637217 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:13.637180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:18.945821 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:18.945787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:19.006451 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:19.006415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:19.092005 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:19.091971    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:18:19.614951 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:19.614920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:25.665405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:25.665376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:25.729374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:25.729343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:25.898962 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:25.898933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:26.165127 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.165090    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"]
Sep 10 23:18:26.169145 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.169125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.172621 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.172600    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5d061fe3-openshift-default-dockercfg-f9pq9\""
Sep 10 23:18:26.178684 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.178456    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"]
Sep 10 23:18:26.191135 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.191101    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:26.293643 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.293603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-token\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.293935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.293910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-credential-socket\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.294116 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.294102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-socket\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.294346 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.294328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcce84-0b87-4f30-87e7-b22c695daaf2-istiod-ca-cert\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.294520 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.294505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-data\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.294728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.294709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwrv\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-kube-api-access-8rwrv\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.294863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.294849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-podinfo\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.295038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.295024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-certs\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.295169 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.295156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-envoy\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.396609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.396567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-certs\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.396824 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.396622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-envoy\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.396824 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.396663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-token\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.396824 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.396680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-credential-socket\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.396824 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.396716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-socket\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.396824 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.396731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcce84-0b87-4f30-87e7-b22c695daaf2-istiod-ca-cert\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.396824 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.396759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-data\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.396824 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.396807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwrv\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-kube-api-access-8rwrv\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.397201 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.396834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-podinfo\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.397201 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.397022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-certs\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.397353 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.397321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-credential-socket\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.397416 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.397381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-socket\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.397513 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.397490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-data\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.397596 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.397496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcce84-0b87-4f30-87e7-b22c695daaf2-istiod-ca-cert\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.399185 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.399152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-envoy\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.399444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.399428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-podinfo\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.404407 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.404375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-token\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.404640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.404609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwrv\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-kube-api-access-8rwrv\") pod \"e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") " pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.429214 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.429122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:26.483294 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.483250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:26.816604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.816572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"]
Sep 10 23:18:26.817723 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:18:26.817697    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fcce84_0b87_4f30_87e7_b22c695daaf2.slice/crio-c5a06d533ac302b4e118b0d5447f0f37ddd0db5bc60f59ad122f5a4e2a7807cf WatchSource:0}: Error finding container c5a06d533ac302b4e118b0d5447f0f37ddd0db5bc60f59ad122f5a4e2a7807cf: Status 404 returned error can't find the container with id c5a06d533ac302b4e118b0d5447f0f37ddd0db5bc60f59ad122f5a4e2a7807cf
Sep 10 23:18:26.820075 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.820042    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:18:26.820162 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.820110    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:18:26.820162 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:26.820138    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:18:27.801945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:27.801913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" event={"ID":"74fcce84-0b87-4f30-87e7-b22c695daaf2","Type":"ContainerStarted","Data":"a7995d765d59946f098aeb892a88265535eaf004b9947df9f0b883db2264070e"}
Sep 10 23:18:27.802133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:27.801965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" event={"ID":"74fcce84-0b87-4f30-87e7-b22c695daaf2","Type":"ContainerStarted","Data":"c5a06d533ac302b4e118b0d5447f0f37ddd0db5bc60f59ad122f5a4e2a7807cf"}
Sep 10 23:18:27.823689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:27.823623    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podStartSLOduration=1.8235968009999999 podStartE2EDuration="1.823596801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:27.821573414 +0000 UTC m=+1007.523206351" watchObservedRunningTime="2026-09-10 23:18:27.823596801 +0000 UTC m=+1007.525229744"
Sep 10 23:18:28.484447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:28.484411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:28.492085 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:28.492043    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:28.807980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:28.807892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:28.808826 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:28.808806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:18:30.702438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:30.702404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:31.869815 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:31.869776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:31.960753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:31.960713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:32.959509 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:32.959474    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"]
Sep 10 23:18:32.964750 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:32.964722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:32.967383 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:32.967352    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5d061fe3-serving-cert\""
Sep 10 23:18:32.967932 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:32.967907    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:18:32.968102 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:32.967804    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5kfc5\""
Sep 10 23:18:32.968180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:32.967595    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:18:32.968921 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:32.968577    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:18:32.971426 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:32.971401    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"]
Sep 10 23:18:32.986978 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:32.986948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:33.067566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.066007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-metrics-certs\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.067566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.066110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-openshift-service-ca\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.067566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.066141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv4wg\" (UniqueName: \"kubernetes.io/projected/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-kube-api-access-zv4wg\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.067566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.066227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-maas-api-tls\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.087993 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.087958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:33.167359 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.167315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-metrics-certs\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.167557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.167402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-openshift-service-ca\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.167557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.167425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv4wg\" (UniqueName: \"kubernetes.io/projected/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-kube-api-access-zv4wg\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.167557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.167481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-maas-api-tls\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.167741 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:18:33.167600    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-serving-cert: secret "maas-api-e2e-mt-5d061fe3-serving-cert" not found
Sep 10 23:18:33.167741 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:18:33.167686    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-maas-api-tls podName:89ded0cd-34a9-46e4-ac9c-3afa3f879b25 nodeName:}" failed. No retries permitted until 2026-09-10 23:18:33.667666162 +0000 UTC m=+1013.369299082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-maas-api-tls") pod "maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4" (UID: "89ded0cd-34a9-46e4-ac9c-3afa3f879b25") : secret "maas-api-e2e-mt-5d061fe3-serving-cert" not found
Sep 10 23:18:33.168125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.168105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-openshift-service-ca\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.170041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.170022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-metrics-certs\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.178135 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.178107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv4wg\" (UniqueName: \"kubernetes.io/projected/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-kube-api-access-zv4wg\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.579784 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.579738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:33.672037 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.671998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-maas-api-tls\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.674705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.674682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-maas-api-tls\") pod \"maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:33.683194 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.683164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:33.879458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:33.879426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:34.157802 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:34.157774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"]
Sep 10 23:18:34.159668 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:18:34.159639    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ded0cd_34a9_46e4_ac9c_3afa3f879b25.slice/crio-08a7c9ffb3f4457a936d354899d103e20603b209ca2ccb74395e16f20e26a3cf WatchSource:0}: Error finding container 08a7c9ffb3f4457a936d354899d103e20603b209ca2ccb74395e16f20e26a3cf: Status 404 returned error can't find the container with id 08a7c9ffb3f4457a936d354899d103e20603b209ca2ccb74395e16f20e26a3cf
Sep 10 23:18:34.161101 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:34.161079    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:18:34.851167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:34.851131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4" event={"ID":"89ded0cd-34a9-46e4-ac9c-3afa3f879b25","Type":"ContainerStarted","Data":"98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3"}
Sep 10 23:18:34.851359 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:34.851186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4" event={"ID":"89ded0cd-34a9-46e4-ac9c-3afa3f879b25","Type":"ContainerStarted","Data":"08a7c9ffb3f4457a936d354899d103e20603b209ca2ccb74395e16f20e26a3cf"}
Sep 10 23:18:34.851359 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:34.851329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:34.870096 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:34.870044    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4" podStartSLOduration=2.494217224 podStartE2EDuration="2.87002823s" totalImagesPullingTime="375.811006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:34.868276671 +0000 UTC m=+1014.569909617" watchObservedRunningTime="2026-09-10 23:18:34.87002823 +0000 UTC m=+1014.571661186"
Sep 10 23:18:40.862363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:40.862337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:18:49.521909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:49.521871    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:18:50.651633 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:50.651598    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:50.709455 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:50.709410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:56.742912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:56.742880    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:57.390444 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:57.390399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:57.485152 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:57.485114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:57.806168 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:57.806055    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:59.779875 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:59.779746    2580 patch_prober.go:28] interesting pod/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": read tcp 10.134.0.2:47356->10.134.0.68:15021: read: connection reset by peer" start-of-body=
Sep 10 23:18:59.779875 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:18:59.779827    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": read tcp 10.134.0.2:47356->10.134.0.68:15021: read: connection reset by peer"
Sep 10 23:19:00.025052 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:00.024971    2580 generic.go:356] "Generic (PLEG): container finished" podID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerID="a7995d765d59946f098aeb892a88265535eaf004b9947df9f0b883db2264070e" exitCode=0
Sep 10 23:19:00.025052 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:00.025040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" event={"ID":"74fcce84-0b87-4f30-87e7-b22c695daaf2","Type":"ContainerDied","Data":"a7995d765d59946f098aeb892a88265535eaf004b9947df9f0b883db2264070e"}
Sep 10 23:19:00.025471 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:00.025457    2580 scope.go:117] "RemoveContainer" containerID="a7995d765d59946f098aeb892a88265535eaf004b9947df9f0b883db2264070e"
Sep 10 23:19:00.034061 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:00.034027    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:19:00.034284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:00.034269    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:19:00.034336 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:00.034303    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:19:01.046132 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:01.046038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" event={"ID":"74fcce84-0b87-4f30-87e7-b22c695daaf2","Type":"ContainerStarted","Data":"757dc3d5c74cf15ff0920cd0489c0aa24c87b5c4d331168daa44c95a1961ba1f"}
Sep 10 23:19:01.483804 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:01.483765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:19:01.484015 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:01.483993    2580 patch_prober.go:28] interesting pod/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 10 23:19:01.484070 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:01.484051    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 10 23:19:02.483835 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:02.483801    2580 patch_prober.go:28] interesting pod/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 10 23:19:02.484245 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:02.483860    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 10 23:19:03.484871 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:03.484837    2580 patch_prober.go:28] interesting pod/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 10 23:19:03.485339 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:03.484900    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 10 23:19:03.489687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:03.489663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:03.588734 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:03.588701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:04.484405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:04.484369    2580 patch_prober.go:28] interesting pod/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 10 23:19:04.484624 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:04.484426    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 10 23:19:05.274318 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:05.274273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:05.484486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:05.484453    2580 patch_prober.go:28] interesting pod/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 10 23:19:05.484687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:05.484506    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 10 23:19:06.484905 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:06.484869    2580 patch_prober.go:28] interesting pod/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 10 23:19:06.485291 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:06.484936    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 10 23:19:07.487636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:07.487604    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:19:07.488135 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:07.488120    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:19:07.488701 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:07.488678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:19:13.184895 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:13.184851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:19:14.343600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.343571    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"]
Sep 10 23:19:14.344001 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.343779    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4" podUID="89ded0cd-34a9-46e4-ac9c-3afa3f879b25" containerName="maas-api" containerID="cri-o://98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3" gracePeriod=30
Sep 10 23:19:14.356441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.356410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:14.424778 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.424745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:14.483597 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.483567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:14.557048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.557009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:14.593799 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.593741    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:19:14.634576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.629956    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-openshift-service-ca\") pod \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") "
Sep 10 23:19:14.634576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.630043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-maas-api-tls\") pod \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") "
Sep 10 23:19:14.634576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.630124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv4wg\" (UniqueName: \"kubernetes.io/projected/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-kube-api-access-zv4wg\") pod \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") "
Sep 10 23:19:14.634576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.630157    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-metrics-certs\") pod \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\" (UID: \"89ded0cd-34a9-46e4-ac9c-3afa3f879b25\") "
Sep 10 23:19:14.641042 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.640198    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89ded0cd-34a9-46e4-ac9c-3afa3f879b25" (UID: "89ded0cd-34a9-46e4-ac9c-3afa3f879b25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:19:14.658354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.658299    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89ded0cd-34a9-46e4-ac9c-3afa3f879b25" (UID: "89ded0cd-34a9-46e4-ac9c-3afa3f879b25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:19:14.659089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.659049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-kube-api-access-zv4wg" (OuterVolumeSpecName: "kube-api-access-zv4wg") pod "89ded0cd-34a9-46e4-ac9c-3afa3f879b25" (UID: "89ded0cd-34a9-46e4-ac9c-3afa3f879b25"). InnerVolumeSpecName "kube-api-access-zv4wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:19:14.666648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.666608    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89ded0cd-34a9-46e4-ac9c-3afa3f879b25" (UID: "89ded0cd-34a9-46e4-ac9c-3afa3f879b25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:19:14.731926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.731895    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:14.731926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.731925    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv4wg\" (UniqueName: \"kubernetes.io/projected/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-kube-api-access-zv4wg\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:14.731926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.731934    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:14.732157 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.731942    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ded0cd-34a9-46e4-ac9c-3afa3f879b25-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:14.886701 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:14.886673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:15.133366 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:15.133329    2580 generic.go:356] "Generic (PLEG): container finished" podID="89ded0cd-34a9-46e4-ac9c-3afa3f879b25" containerID="98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3" exitCode=0
Sep 10 23:19:15.133366 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:15.133367    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4" event={"ID":"89ded0cd-34a9-46e4-ac9c-3afa3f879b25","Type":"ContainerDied","Data":"98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3"}
Sep 10 23:19:15.133636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:15.133388    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"
Sep 10 23:19:15.133636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:15.133406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4" event={"ID":"89ded0cd-34a9-46e4-ac9c-3afa3f879b25","Type":"ContainerDied","Data":"08a7c9ffb3f4457a936d354899d103e20603b209ca2ccb74395e16f20e26a3cf"}
Sep 10 23:19:15.133636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:15.133427    2580 scope.go:117] "RemoveContainer" containerID="98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3"
Sep 10 23:19:15.143106 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:15.143087    2580 scope.go:117] "RemoveContainer" containerID="98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3"
Sep 10 23:19:15.143402 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:19:15.143380    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3\": container with ID starting with 98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3 not found: ID does not exist" containerID="98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3"
Sep 10 23:19:15.143446 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:15.143411    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3"} err="failed to get container status \"98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3\": rpc error: code = NotFound desc = could not find container \"98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3\": container with ID starting with 98b1d0b64b8fefb7be027da30a97b08abe7bd288687e8fa04201560145b879c3 not found: ID does not exist"
Sep 10 23:19:15.154405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:15.154378    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"]
Sep 10 23:19:15.157218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:15.157198    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d061fe3-7b44b9fcb9-zfgp4"]
Sep 10 23:19:16.834890 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:16.834852    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89ded0cd-34a9-46e4-ac9c-3afa3f879b25" path="/var/lib/kubelet/pods/89ded0cd-34a9-46e4-ac9c-3afa3f879b25/volumes"
Sep 10 23:19:19.165408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:19.165381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:20.764278 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:20.764243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:20.821514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:20.821486    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:21.003723 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:21.003694    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"]
Sep 10 23:19:21.003919 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:21.003898    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy" containerID="cri-o://757dc3d5c74cf15ff0920cd0489c0aa24c87b5c4d331168daa44c95a1961ba1f" gracePeriod=30
Sep 10 23:19:21.021147 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:21.021072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:21.350158 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:21.350124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:21.448744 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:21.448707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:21.549396 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:21.549362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:22.490678 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:22.490646    2580 patch_prober.go:28] interesting pod/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 10 23:19:22.491125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:22.490742    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:19:26.202053 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.202023    2580 generic.go:356] "Generic (PLEG): container finished" podID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerID="757dc3d5c74cf15ff0920cd0489c0aa24c87b5c4d331168daa44c95a1961ba1f" exitCode=0
Sep 10 23:19:26.202438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.202089    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" event={"ID":"74fcce84-0b87-4f30-87e7-b22c695daaf2","Type":"ContainerDied","Data":"757dc3d5c74cf15ff0920cd0489c0aa24c87b5c4d331168daa44c95a1961ba1f"}
Sep 10 23:19:26.202438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.202171    2580 scope.go:117] "RemoveContainer" containerID="a7995d765d59946f098aeb892a88265535eaf004b9947df9f0b883db2264070e"
Sep 10 23:19:26.261506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.261482    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:19:26.349131 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-data\") pod \"74fcce84-0b87-4f30-87e7-b22c695daaf2\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") "
Sep 10 23:19:26.349314 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-podinfo\") pod \"74fcce84-0b87-4f30-87e7-b22c695daaf2\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") "
Sep 10 23:19:26.349314 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-envoy\") pod \"74fcce84-0b87-4f30-87e7-b22c695daaf2\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") "
Sep 10 23:19:26.349314 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-token\") pod \"74fcce84-0b87-4f30-87e7-b22c695daaf2\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") "
Sep 10 23:19:26.349314 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rwrv\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-kube-api-access-8rwrv\") pod \"74fcce84-0b87-4f30-87e7-b22c695daaf2\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") "
Sep 10 23:19:26.349573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcce84-0b87-4f30-87e7-b22c695daaf2-istiod-ca-cert\") pod \"74fcce84-0b87-4f30-87e7-b22c695daaf2\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") "
Sep 10 23:19:26.349573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349365    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-socket\") pod \"74fcce84-0b87-4f30-87e7-b22c695daaf2\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") "
Sep 10 23:19:26.349573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349388    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-credential-socket\") pod \"74fcce84-0b87-4f30-87e7-b22c695daaf2\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") "
Sep 10 23:19:26.349573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-data" (OuterVolumeSpecName: "istio-data") pod "74fcce84-0b87-4f30-87e7-b22c695daaf2" (UID: "74fcce84-0b87-4f30-87e7-b22c695daaf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:19:26.349573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349417    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-certs\") pod \"74fcce84-0b87-4f30-87e7-b22c695daaf2\" (UID: \"74fcce84-0b87-4f30-87e7-b22c695daaf2\") "
Sep 10 23:19:26.349819 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349665    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74fcce84-0b87-4f30-87e7-b22c695daaf2" (UID: "74fcce84-0b87-4f30-87e7-b22c695daaf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:19:26.349872 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74fcce84-0b87-4f30-87e7-b22c695daaf2" (UID: "74fcce84-0b87-4f30-87e7-b22c695daaf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:19:26.349872 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349843    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74fcce84-0b87-4f30-87e7-b22c695daaf2" (UID: "74fcce84-0b87-4f30-87e7-b22c695daaf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:19:26.349872 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.349847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fcce84-0b87-4f30-87e7-b22c695daaf2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74fcce84-0b87-4f30-87e7-b22c695daaf2" (UID: "74fcce84-0b87-4f30-87e7-b22c695daaf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:19:26.350028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.350006    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:26.350140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.350033    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:26.350140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.350048    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:26.350140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.350061    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcce84-0b87-4f30-87e7-b22c695daaf2-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:26.350140 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.350073    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:26.351719 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.351700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74fcce84-0b87-4f30-87e7-b22c695daaf2" (UID: "74fcce84-0b87-4f30-87e7-b22c695daaf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:19:26.351902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.351879    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-kube-api-access-8rwrv" (OuterVolumeSpecName: "kube-api-access-8rwrv") pod "74fcce84-0b87-4f30-87e7-b22c695daaf2" (UID: "74fcce84-0b87-4f30-87e7-b22c695daaf2"). InnerVolumeSpecName "kube-api-access-8rwrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:19:26.352018 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.352001    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-token" (OuterVolumeSpecName: "istio-token") pod "74fcce84-0b87-4f30-87e7-b22c695daaf2" (UID: "74fcce84-0b87-4f30-87e7-b22c695daaf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:19:26.352076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.352012    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74fcce84-0b87-4f30-87e7-b22c695daaf2" (UID: "74fcce84-0b87-4f30-87e7-b22c695daaf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:19:26.451653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.451622    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:26.451653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.451646    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:26.451653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.451654    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:26.451871 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:26.451663    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rwrv\" (UniqueName: \"kubernetes.io/projected/74fcce84-0b87-4f30-87e7-b22c695daaf2-kube-api-access-8rwrv\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:27.209174 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:27.209089    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"
Sep 10 23:19:27.209174 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:27.209124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5" event={"ID":"74fcce84-0b87-4f30-87e7-b22c695daaf2","Type":"ContainerDied","Data":"c5a06d533ac302b4e118b0d5447f0f37ddd0db5bc60f59ad122f5a4e2a7807cf"}
Sep 10 23:19:27.209691 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:27.209182    2580 scope.go:117] "RemoveContainer" containerID="757dc3d5c74cf15ff0920cd0489c0aa24c87b5c4d331168daa44c95a1961ba1f"
Sep 10 23:19:27.228940 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:27.228914    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"]
Sep 10 23:19:27.232205 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:27.232185    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5d061fe3-openshift-default-79b45ccdb4-dm8j5"]
Sep 10 23:19:27.381716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:27.381687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:27.447811 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:27.447778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:27.520991 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:27.520899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:27.701133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:27.701089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:28.834872 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:28.834834    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" path="/var/lib/kubelet/pods/74fcce84-0b87-4f30-87e7-b22c695daaf2/volumes"
Sep 10 23:19:33.420973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:33.420884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:33.482938 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:33.482905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:33.774870 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:33.774780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:34.289524 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.289488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:34.557200 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557103    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"]
Sep 10 23:19:34.557735 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557719    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy"
Sep 10 23:19:34.557790 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557739    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy"
Sep 10 23:19:34.557790 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557754    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy"
Sep 10 23:19:34.557790 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557762    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy"
Sep 10 23:19:34.557892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557788    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ded0cd-34a9-46e4-ac9c-3afa3f879b25" containerName="maas-api"
Sep 10 23:19:34.557892 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557797    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ded0cd-34a9-46e4-ac9c-3afa3f879b25" containerName="maas-api"
Sep 10 23:19:34.557958 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557896    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy"
Sep 10 23:19:34.557958 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557909    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fcce84-0b87-4f30-87e7-b22c695daaf2" containerName="istio-proxy"
Sep 10 23:19:34.557958 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.557919    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ded0cd-34a9-46e4-ac9c-3afa3f879b25" containerName="maas-api"
Sep 10 23:19:34.561495 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.561473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.563760 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.563735    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-bda598-openshift-default-dockercfg-qh7rx\""
Sep 10 23:19:34.572086 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.572059    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"]
Sep 10 23:19:34.583286 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.583257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:34.682902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.682872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:34.739176 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.739138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-envoy\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.739492 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.739470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-socket\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.739670 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.739637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istiod-ca-cert\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.739803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.739689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-podinfo\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.739803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.739765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-token\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.739803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.739794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-certs\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.739979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.739826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-credential-socket\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.739979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.739866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-data\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.739979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.739893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n44sr\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-kube-api-access-n44sr\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-token\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-certs\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-credential-socket\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-data\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n44sr\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-kube-api-access-n44sr\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-envoy\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-socket\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istiod-ca-cert\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-podinfo\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841910 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-credential-socket\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.841968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-certs\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.842011 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.841978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-data\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.842046 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.842008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-socket\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.842227 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.842209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istiod-ca-cert\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.843860 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.843838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-envoy\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.844104 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.844086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-podinfo\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.848118 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.848100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-token\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.848205 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.848175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n44sr\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-kube-api-access-n44sr\") pod \"e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") " pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.873966 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.873942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:34.874486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:34.874465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:35.153354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:35.153324    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"]
Sep 10 23:19:35.154883 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:19:35.154853    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a7d0cd_a7db_4fe3_991c_3e4e5959b43d.slice/crio-a7ce587448fbeadb2f6c84ee1095b6a4e1c14f18e2082eb210f4392f94692ed4 WatchSource:0}: Error finding container a7ce587448fbeadb2f6c84ee1095b6a4e1c14f18e2082eb210f4392f94692ed4: Status 404 returned error can't find the container with id a7ce587448fbeadb2f6c84ee1095b6a4e1c14f18e2082eb210f4392f94692ed4
Sep 10 23:19:35.157454 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:35.157417    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:19:35.157592 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:35.157501    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:19:35.157592 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:35.157567    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:19:35.261531 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:35.261475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" event={"ID":"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d","Type":"ContainerStarted","Data":"5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a"}
Sep 10 23:19:35.261744 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:35.261560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" event={"ID":"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d","Type":"ContainerStarted","Data":"a7ce587448fbeadb2f6c84ee1095b6a4e1c14f18e2082eb210f4392f94692ed4"}
Sep 10 23:19:35.280524 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:35.280474    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" podStartSLOduration=1.280459486 podStartE2EDuration="1.280459486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:19:35.27817249 +0000 UTC m=+1074.979805433" watchObservedRunningTime="2026-09-10 23:19:35.280459486 +0000 UTC m=+1074.982092430"
Sep 10 23:19:35.874076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:35.874042    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:36.880068 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:36.880040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:37.141072 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.140991    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:37.231189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.231158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:37.273559 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.273512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:37.274557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.274521    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:19:37.462847 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.462765    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:37.862554 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.862505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"]
Sep 10 23:19:37.870494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.867032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"
Sep 10 23:19:37.871942 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.871917    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:19:37.872112 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.871917    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:19:37.875123 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.875097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"]
Sep 10 23:19:37.976908 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:37.976871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1268d-050b-4b7c-a935-a5f6475077e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww\" (UID: \"08b1268d-050b-4b7c-a935-a5f6475077e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"
Sep 10 23:19:38.077645 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:38.077611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1268d-050b-4b7c-a935-a5f6475077e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww\" (UID: \"08b1268d-050b-4b7c-a935-a5f6475077e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"
Sep 10 23:19:38.078258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:38.078235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1268d-050b-4b7c-a935-a5f6475077e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww\" (UID: \"08b1268d-050b-4b7c-a935-a5f6475077e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"
Sep 10 23:19:38.181893 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:38.181814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"
Sep 10 23:19:38.388252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:38.388222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:38.453525 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:38.453431    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"]
Sep 10 23:19:39.176590 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.176558    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"]
Sep 10 23:19:39.179357 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.179332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.182215 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.181987    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:19:39.182215 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.182021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:19:39.182215 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.182021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5kfc5\""
Sep 10 23:19:39.182454 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.182082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-bda598-serving-cert\""
Sep 10 23:19:39.192452 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.192427    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"]
Sep 10 23:19:39.205593 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.205535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:39.293580 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.293249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.293580 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.293320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.293580 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.293512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.293580 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.293567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsx8\" (UniqueName: \"kubernetes.io/projected/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-kube-api-access-kbsx8\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.298715 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.298676    2580 generic.go:356] "Generic (PLEG): container finished" podID="08b1268d-050b-4b7c-a935-a5f6475077e4" containerID="51b96c1ba4bf1a3e50c23bdedeed692b8d144bc2fd8881120d92c9477f937ae1" exitCode=0
Sep 10 23:19:39.298916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.298826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww" event={"ID":"08b1268d-050b-4b7c-a935-a5f6475077e4","Type":"ContainerDied","Data":"51b96c1ba4bf1a3e50c23bdedeed692b8d144bc2fd8881120d92c9477f937ae1"}
Sep 10 23:19:39.298916 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.298900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww" event={"ID":"08b1268d-050b-4b7c-a935-a5f6475077e4","Type":"ContainerStarted","Data":"a3e0853a016466764005b200494715fd24838f045ca0ab784d37a746bd57308c"}
Sep 10 23:19:39.306103 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.306080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:39.394300 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.394266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.394300 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.394300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsx8\" (UniqueName: \"kubernetes.io/projected/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-kube-api-access-kbsx8\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.394567 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.394343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.394567 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.394375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.394567 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:19:39.394508    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-serving-cert: secret "maas-api-e2e-shared-a-w2-bda598-serving-cert" not found
Sep 10 23:19:39.394720 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:19:39.394597    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-maas-api-tls podName:adcea5dc-ad08-451d-b5ae-01ea0b6da77b nodeName:}" failed. No retries permitted until 2026-09-10 23:19:39.894575491 +0000 UTC m=+1079.596208427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj" (UID: "adcea5dc-ad08-451d-b5ae-01ea0b6da77b") : secret "maas-api-e2e-shared-a-w2-bda598-serving-cert" not found
Sep 10 23:19:39.395079 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.395061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.396823 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.396800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.402313 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.402293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsx8\" (UniqueName: \"kubernetes.io/projected/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-kube-api-access-kbsx8\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.638446 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.638406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:39.735675 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.735646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:39.801749 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.801713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:39.901265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.901131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:39.904596 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:39.904530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:40.096283 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:40.096247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:40.376002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:40.375969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"]
Sep 10 23:19:40.377696 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:19:40.377669    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcea5dc_ad08_451d_b5ae_01ea0b6da77b.slice/crio-c493228485271475d012d99bd9b0ffafdac5e4a8ea11b16e26f4438d784974b3 WatchSource:0}: Error finding container c493228485271475d012d99bd9b0ffafdac5e4a8ea11b16e26f4438d784974b3: Status 404 returned error can't find the container with id c493228485271475d012d99bd9b0ffafdac5e4a8ea11b16e26f4438d784974b3
Sep 10 23:19:40.441322 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:40.441298    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"
Sep 10 23:19:40.507465 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:40.507428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1268d-050b-4b7c-a935-a5f6475077e4-maas-api-service-ca\") pod \"08b1268d-050b-4b7c-a935-a5f6475077e4\" (UID: \"08b1268d-050b-4b7c-a935-a5f6475077e4\") "
Sep 10 23:19:40.507827 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:40.507804    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b1268d-050b-4b7c-a935-a5f6475077e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08b1268d-050b-4b7c-a935-a5f6475077e4" (UID: "08b1268d-050b-4b7c-a935-a5f6475077e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:19:40.608345 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:40.608311    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1268d-050b-4b7c-a935-a5f6475077e4-maas-api-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:19:41.312016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:41.311937    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"
Sep 10 23:19:41.312016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:41.311979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww" event={"ID":"08b1268d-050b-4b7c-a935-a5f6475077e4","Type":"ContainerDied","Data":"a3e0853a016466764005b200494715fd24838f045ca0ab784d37a746bd57308c"}
Sep 10 23:19:41.312249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:41.312025    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e0853a016466764005b200494715fd24838f045ca0ab784d37a746bd57308c"
Sep 10 23:19:41.313631 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:41.313604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj" event={"ID":"adcea5dc-ad08-451d-b5ae-01ea0b6da77b","Type":"ContainerStarted","Data":"7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb"}
Sep 10 23:19:41.313742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:41.313651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj" event={"ID":"adcea5dc-ad08-451d-b5ae-01ea0b6da77b","Type":"ContainerStarted","Data":"c493228485271475d012d99bd9b0ffafdac5e4a8ea11b16e26f4438d784974b3"}
Sep 10 23:19:41.313849 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:41.313829    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:41.331634 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:41.331593    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj" podStartSLOduration=1.9494296420000001 podStartE2EDuration="2.331579503s" totalImagesPullingTime="382.149861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:19:41.328805494 +0000 UTC m=+1081.030438437" watchObservedRunningTime="2026-09-10 23:19:41.331579503 +0000 UTC m=+1081.033212446"
Sep 10 23:19:42.573408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:42.573379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:19:47.326712 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:47.326685    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:19:47.994125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:47.994096    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:48.559576 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:48.559525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:48.658744 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:48.658711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:48.737849 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:48.737821    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"]
Sep 10 23:19:48.739431 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:48.739411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:48.740563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:48.740528    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b3d9be-rdsww"]
Sep 10 23:19:48.793626 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:48.793595    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:48.854589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:48.853534    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b1268d-050b-4b7c-a935-a5f6475077e4" path="/var/lib/kubelet/pods/08b1268d-050b-4b7c-a935-a5f6475077e4/volumes"
Sep 10 23:19:49.104693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:49.104607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:54.115659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:54.115622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:59.081355 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:19:59.081324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:00.050478 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:00.050444    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:02.054950 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.054913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:02.370611 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.370572    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:02.638623 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.634322    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"]
Sep 10 23:20:02.638623 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.635267    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b1268d-050b-4b7c-a935-a5f6475077e4" containerName="revoke-keys"
Sep 10 23:20:02.638623 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.635287    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b1268d-050b-4b7c-a935-a5f6475077e4" containerName="revoke-keys"
Sep 10 23:20:02.642563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.642324    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b1268d-050b-4b7c-a935-a5f6475077e4" containerName="revoke-keys"
Sep 10 23:20:02.648657 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.646979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.648657 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.648275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:02.650617 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.650032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-7b202f-openshift-default-dockercfg-2gwgg\""
Sep 10 23:20:02.663566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.663499    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"]
Sep 10 23:20:02.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.744164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d952917-3a68-4659-8674-52eb7a85f4b3-istiod-ca-cert\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.744221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8k6c\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-kube-api-access-t8k6c\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.744272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-socket\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.744318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-podinfo\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.744380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-envoy\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.744415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-certs\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.744448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-token\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.744490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-credential-socket\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.745566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.744530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-data\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.846101 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.846061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-certs\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.847294 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.847265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-token\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.847473 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.847446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-certs\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.847642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.847621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-credential-socket\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.847797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.847781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-data\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.848077 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.848035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d952917-3a68-4659-8674-52eb7a85f4b3-istiod-ca-cert\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.848239 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.848221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8k6c\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-kube-api-access-t8k6c\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.848389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.848373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-socket\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.848555 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.848519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-podinfo\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.848709 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.848684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-credential-socket\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.848864 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.848697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d952917-3a68-4659-8674-52eb7a85f4b3-istiod-ca-cert\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.848968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.848709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-envoy\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.849112 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.848298    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-data\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.849224 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.848797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-socket\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.851423 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.851398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-envoy\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.852016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.851995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-podinfo\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.857359 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.857327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-token\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.857791 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.857770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8k6c\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-kube-api-access-t8k6c\") pod \"e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") " pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:02.966607 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:02.966496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:03.265030 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.265000    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"]
Sep 10 23:20:03.266702 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:20:03.266675    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d952917_3a68_4659_8674_52eb7a85f4b3.slice/crio-074138bd4a6bf4ba973dd9418278bfe988a18d4551f1f9ef7e8eb4876926d91a WatchSource:0}: Error finding container 074138bd4a6bf4ba973dd9418278bfe988a18d4551f1f9ef7e8eb4876926d91a: Status 404 returned error can't find the container with id 074138bd4a6bf4ba973dd9418278bfe988a18d4551f1f9ef7e8eb4876926d91a
Sep 10 23:20:03.269453 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.269417    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:03.269563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.269496    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:03.269635 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.269557    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:03.460319 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.460277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" event={"ID":"3d952917-3a68-4659-8674-52eb7a85f4b3","Type":"ContainerStarted","Data":"6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6"}
Sep 10 23:20:03.460519 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.460352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" event={"ID":"3d952917-3a68-4659-8674-52eb7a85f4b3","Type":"ContainerStarted","Data":"074138bd4a6bf4ba973dd9418278bfe988a18d4551f1f9ef7e8eb4876926d91a"}
Sep 10 23:20:03.482596 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.482517    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" podStartSLOduration=1.4824956839999999 podStartE2EDuration="1.482495684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:03.479647581 +0000 UTC m=+1103.181280525" watchObservedRunningTime="2026-09-10 23:20:03.482495684 +0000 UTC m=+1103.184128629"
Sep 10 23:20:03.552777 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.552687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:03.877660 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.877632    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:03.966661 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:03.966635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:04.972078 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:04.972049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:05.361861 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:05.361816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:05.460462 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:05.460424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:05.472990 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:05.472962    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:05.474089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:05.474068    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:06.586463 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:06.586431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:06.678127 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:06.678099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:07.057838 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:07.057806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:07.155869 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:07.155832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:08.321740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:08.321700    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 23:20:08.322205 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:08.321780    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 23:20:10.508899 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:10.508866    2580 generic.go:356] "Generic (PLEG): container finished" podID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerID="5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a" exitCode=0
Sep 10 23:20:10.509339 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:10.508946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" event={"ID":"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d","Type":"ContainerDied","Data":"5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a"}
Sep 10 23:20:10.509384 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:10.509363    2580 scope.go:117] "RemoveContainer" containerID="5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a"
Sep 10 23:20:10.510534 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:10.510506    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:10.510612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:10.510594    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:10.510648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:10.510627    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:11.516350 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:11.516311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" event={"ID":"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d","Type":"ContainerStarted","Data":"425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5"}
Sep 10 23:20:11.874598 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:11.874560    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:20:11.874801 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:11.874779    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 23:20:11.874857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:11.874830    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 23:20:12.874339 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:12.874305    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 23:20:12.874787 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:12.874369    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 23:20:13.874305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:13.874262    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 23:20:13.874522 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:13.874325    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 23:20:14.875303 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:14.875269    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 23:20:14.875711 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:14.875338    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 23:20:15.079067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:15.079034    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:20:15.875132 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:15.875095    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 23:20:15.875320 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:15.875145    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 23:20:16.875142 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:16.875109    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 23:20:16.875518 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:16.875166    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 23:20:17.895527 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:17.895492    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:20:17.895968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:17.895906    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:20:17.896746 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:17.896724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:20:27.638703 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:27.638669    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerID="6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6" exitCode=0
Sep 10 23:20:27.639108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:27.638753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" event={"ID":"3d952917-3a68-4659-8674-52eb7a85f4b3","Type":"ContainerDied","Data":"6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6"}
Sep 10 23:20:27.639219 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:27.639206    2580 scope.go:117] "RemoveContainer" containerID="6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6"
Sep 10 23:20:27.640992 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:27.640943    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:27.641290 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:27.641258    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:27.641484 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:27.641459    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:28.649134 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:28.649099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" event={"ID":"3d952917-3a68-4659-8674-52eb7a85f4b3","Type":"ContainerStarted","Data":"a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b"}
Sep 10 23:20:28.967786 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:28.967690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:28.967968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:28.967926    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 23:20:28.968043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:28.967999    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 23:20:29.967968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:29.967930    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 23:20:29.968366 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:29.968000    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 23:20:30.274304 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:30.274224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:30.967164 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:30.967125    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 23:20:30.967347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:30.967187    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 23:20:31.457752 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:31.457719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:31.679100 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:31.679065    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"]
Sep 10 23:20:31.679336 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:31.679314    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy" containerID="cri-o://28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50" gracePeriod=30
Sep 10 23:20:31.693337 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:31.693312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:31.967868 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:31.967836    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 23:20:31.968066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:31.967901    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 23:20:32.181464 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.181430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:32.387521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.387481    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"]
Sep 10 23:20:32.390791 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.390767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.393321 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.393299    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-c93e93-openshift-default-dockercfg-cvbls\""
Sep 10 23:20:32.403562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.403521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"]
Sep 10 23:20:32.407680 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.407657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:32.478788 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.478750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-data\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.479273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.478822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcxl\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-kube-api-access-fzcxl\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.479273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.478879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-credential-socket\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.479273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.478948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-envoy\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.479273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.479013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-certs\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.479273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.479057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-podinfo\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.479273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.479191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-socket\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.479273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.479235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-token\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.479273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.479274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istiod-ca-cert\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.580780 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.580744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-socket\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.580935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.580801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-token\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.580935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.580838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istiod-ca-cert\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.580935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.580931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-data\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581059 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.580976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcxl\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-kube-api-access-fzcxl\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581059 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.581029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-credential-socket\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581126 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.581099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-envoy\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581163 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.581125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-certs\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581163 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.581154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-podinfo\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581255 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.581234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-socket\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581598 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.581318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-data\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581598 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.581488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-credential-socket\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.581608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-certs\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.581865 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.581847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istiod-ca-cert\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.584326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.584306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-envoy\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.584473 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.584404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-podinfo\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.589497 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.589468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-token\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.589638 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.589558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcxl\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-kube-api-access-fzcxl\") pod \"e2e-noten-c93e93-openshift-default-86965d8768-cf2rq\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") " pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.704302 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.704215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:32.967708 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.967615    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 23:20:32.967708 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:32.967680    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 23:20:33.012703 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.012678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"]
Sep 10 23:20:33.016247 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:20:33.016198    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d779c6_ebd2_4487_b8f7_6f9f6aa2c100.slice/crio-749d6eedea8f33ff29d0a9edcba67ea6ade28678104975914bfdfb38a9f27452 WatchSource:0}: Error finding container 749d6eedea8f33ff29d0a9edcba67ea6ade28678104975914bfdfb38a9f27452: Status 404 returned error can't find the container with id 749d6eedea8f33ff29d0a9edcba67ea6ade28678104975914bfdfb38a9f27452
Sep 10 23:20:33.024751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.022897    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:33.024751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.022981    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:33.024751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.023024    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:33.109400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.109371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:33.682691 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.682660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq" event={"ID":"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100","Type":"ContainerStarted","Data":"13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62"}
Sep 10 23:20:33.683093 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.682712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq" event={"ID":"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100","Type":"ContainerStarted","Data":"749d6eedea8f33ff29d0a9edcba67ea6ade28678104975914bfdfb38a9f27452"}
Sep 10 23:20:33.701956 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.701904    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq" podStartSLOduration=1.701887374 podStartE2EDuration="1.701887374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:33.700736253 +0000 UTC m=+1133.402369197" watchObservedRunningTime="2026-09-10 23:20:33.701887374 +0000 UTC m=+1133.403520317"
Sep 10 23:20:33.704751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.704728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:33.964228 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:33.964142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:34.033008 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:34.032977    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:34.033535 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:34.033474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:34.035009 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:34.034989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:20:34.056333 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:34.056308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:34.212748 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:34.212715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:34.710193 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:34.710165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:35.695785 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:35.695748    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:35.696717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:35.696691    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:20:36.939063 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:36.939027    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:20:37.032347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-credential-socket\") pod \"8588db66-e7c9-4060-b143-8d9c05e8315a\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") "
Sep 10 23:20:37.032347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032323    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnpgq\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-kube-api-access-hnpgq\") pod \"8588db66-e7c9-4060-b143-8d9c05e8315a\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") "
Sep 10 23:20:37.032612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032352    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-token\") pod \"8588db66-e7c9-4060-b143-8d9c05e8315a\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") "
Sep 10 23:20:37.032612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-envoy\") pod \"8588db66-e7c9-4060-b143-8d9c05e8315a\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") "
Sep 10 23:20:37.032612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032389    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8588db66-e7c9-4060-b143-8d9c05e8315a-istiod-ca-cert\") pod \"8588db66-e7c9-4060-b143-8d9c05e8315a\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") "
Sep 10 23:20:37.032612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032452    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-certs\") pod \"8588db66-e7c9-4060-b143-8d9c05e8315a\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") "
Sep 10 23:20:37.032612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032481    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-podinfo\") pod \"8588db66-e7c9-4060-b143-8d9c05e8315a\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") "
Sep 10 23:20:37.032612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8588db66-e7c9-4060-b143-8d9c05e8315a" (UID: "8588db66-e7c9-4060-b143-8d9c05e8315a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:20:37.032934 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032666    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-socket\") pod \"8588db66-e7c9-4060-b143-8d9c05e8315a\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") "
Sep 10 23:20:37.032934 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-data\") pod \"8588db66-e7c9-4060-b143-8d9c05e8315a\" (UID: \"8588db66-e7c9-4060-b143-8d9c05e8315a\") "
Sep 10 23:20:37.032934 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032813    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8588db66-e7c9-4060-b143-8d9c05e8315a" (UID: "8588db66-e7c9-4060-b143-8d9c05e8315a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:20:37.032934 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.032867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8588db66-e7c9-4060-b143-8d9c05e8315a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8588db66-e7c9-4060-b143-8d9c05e8315a" (UID: "8588db66-e7c9-4060-b143-8d9c05e8315a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:20:37.033312 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.033009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8588db66-e7c9-4060-b143-8d9c05e8315a" (UID: "8588db66-e7c9-4060-b143-8d9c05e8315a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:20:37.033378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.033355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8588db66-e7c9-4060-b143-8d9c05e8315a" (UID: "8588db66-e7c9-4060-b143-8d9c05e8315a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:20:37.033471 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.033449    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.033531 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.033476    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.033531 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.033492    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.033531 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.033506    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8588db66-e7c9-4060-b143-8d9c05e8315a-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.033531 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.033520    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.034849 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.034828    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8588db66-e7c9-4060-b143-8d9c05e8315a" (UID: "8588db66-e7c9-4060-b143-8d9c05e8315a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:20:37.034961 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.034904    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-kube-api-access-hnpgq" (OuterVolumeSpecName: "kube-api-access-hnpgq") pod "8588db66-e7c9-4060-b143-8d9c05e8315a" (UID: "8588db66-e7c9-4060-b143-8d9c05e8315a"). InnerVolumeSpecName "kube-api-access-hnpgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:37.034961 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.034915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8588db66-e7c9-4060-b143-8d9c05e8315a" (UID: "8588db66-e7c9-4060-b143-8d9c05e8315a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:37.035060 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.035041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8588db66-e7c9-4060-b143-8d9c05e8315a" (UID: "8588db66-e7c9-4060-b143-8d9c05e8315a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:20:37.134438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.134382    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpgq\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-kube-api-access-hnpgq\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.134438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.134433    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.134438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.134446    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.134791 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.134458    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8588db66-e7c9-4060-b143-8d9c05e8315a-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.709079 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.709049    2580 generic.go:356] "Generic (PLEG): container finished" podID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerID="28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50" exitCode=0
Sep 10 23:20:37.709249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.709099    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"
Sep 10 23:20:37.709249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.709133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" event={"ID":"8588db66-e7c9-4060-b143-8d9c05e8315a","Type":"ContainerDied","Data":"28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50"}
Sep 10 23:20:37.709249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.709184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p" event={"ID":"8588db66-e7c9-4060-b143-8d9c05e8315a","Type":"ContainerDied","Data":"ec9bd0d8d593a6fc9d1e1a2db94ef73f8d1951212b30ed06eed4e92084e8d44e"}
Sep 10 23:20:37.709249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.709206    2580 scope.go:117] "RemoveContainer" containerID="28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50"
Sep 10 23:20:37.721256 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.721240    2580 scope.go:117] "RemoveContainer" containerID="77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33"
Sep 10 23:20:37.730478 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.730459    2580 scope.go:117] "RemoveContainer" containerID="28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50"
Sep 10 23:20:37.730814 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:20:37.730791    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50\": container with ID starting with 28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50 not found: ID does not exist" containerID="28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50"
Sep 10 23:20:37.730901 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.730819    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50"} err="failed to get container status \"28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50\": rpc error: code = NotFound desc = could not find container \"28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50\": container with ID starting with 28ec38b81a600d608853912f31ddf1632d079bde526d0d93c072317e27c8da50 not found: ID does not exist"
Sep 10 23:20:37.730901 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.730837    2580 scope.go:117] "RemoveContainer" containerID="77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33"
Sep 10 23:20:37.731094 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:20:37.731075    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33\": container with ID starting with 77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33 not found: ID does not exist" containerID="77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33"
Sep 10 23:20:37.731136 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.731100    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33"} err="failed to get container status \"77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33\": rpc error: code = NotFound desc = could not find container \"77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33\": container with ID starting with 77288c47fb2ff4770627e06e466cdfbcb8419f488110b8f7dfe5075115e9ff33 not found: ID does not exist"
Sep 10 23:20:37.731772 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.731749    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"]
Sep 10 23:20:37.735201 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:37.735183    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b3d9be-openshift-default-76d446df4d-w2f9p"]
Sep 10 23:20:38.834011 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:38.833975    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" path="/var/lib/kubelet/pods/8588db66-e7c9-4060-b143-8d9c05e8315a/volumes"
Sep 10 23:20:39.835600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:39.835565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:40.105594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.105564    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"]
Sep 10 23:20:40.106183 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.106166    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy"
Sep 10 23:20:40.106252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.106186    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy"
Sep 10 23:20:40.106252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.106194    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy"
Sep 10 23:20:40.106252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.106199    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy"
Sep 10 23:20:40.106421 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.106298    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy"
Sep 10 23:20:40.106421 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.106309    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8588db66-e7c9-4060-b143-8d9c05e8315a" containerName="istio-proxy"
Sep 10 23:20:40.108661 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.108631    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.110646 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.110618    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-76145153-openshift-default-dockercfg-xvwbz\""
Sep 10 23:20:40.121494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.121472    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"]
Sep 10 23:20:40.128287 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.128243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:40.269181 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.269140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-socket\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.269389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.269195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-certs\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.269389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.269219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istiod-ca-cert\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.269389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.269289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-envoy\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.269389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.269345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-podinfo\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.269389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.269364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-token\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.269389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.269383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-data\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.269748 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.269424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-credential-socket\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.269748 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.269464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlpn6\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-kube-api-access-qlpn6\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.370777 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.370685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-socket\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.370777 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.370751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-certs\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.370994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.370784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istiod-ca-cert\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.370994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.370831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-envoy\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.370994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.370892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-podinfo\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.370994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.370921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-token\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.370994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.370951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-data\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.370994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.370980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-credential-socket\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.371299 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.371021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpn6\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-kube-api-access-qlpn6\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.371299 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.371266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-socket\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.371510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.371484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-data\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.371586 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.371536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istiod-ca-cert\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.371647 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.371588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-certs\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.371705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.371683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-credential-socket\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.373427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.373397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-envoy\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.373661 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.373641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-podinfo\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.379498 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.379476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlpn6\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-kube-api-access-qlpn6\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.379829 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.379807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-token\") pod \"e2e-mt-76145153-openshift-default-979f65fcb-q7g24\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") " pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.426012 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.425978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:40.429328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.429210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:40.700857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.700833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"]
Sep 10 23:20:40.703287 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:20:40.703229    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1673bcae_c356_4d22_9f5a_5ec8a515d22b.slice/crio-710d8894dc667ead774a80d79096b17fab2a0717cfec656077e5d417a1379160 WatchSource:0}: Error finding container 710d8894dc667ead774a80d79096b17fab2a0717cfec656077e5d417a1379160: Status 404 returned error can't find the container with id 710d8894dc667ead774a80d79096b17fab2a0717cfec656077e5d417a1379160
Sep 10 23:20:40.710455 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.710184    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:40.710638 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.710591    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:40.710899 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.710871    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:20:40.731953 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:40.731921    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" event={"ID":"1673bcae-c356-4d22-9f5a-5ec8a515d22b","Type":"ContainerStarted","Data":"710d8894dc667ead774a80d79096b17fab2a0717cfec656077e5d417a1379160"}
Sep 10 23:20:41.740314 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:41.740275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" event={"ID":"1673bcae-c356-4d22-9f5a-5ec8a515d22b","Type":"ContainerStarted","Data":"a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce"}
Sep 10 23:20:41.760505 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:41.760442    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" podStartSLOduration=1.760419419 podStartE2EDuration="1.760419419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:41.757999115 +0000 UTC m=+1141.459632070" watchObservedRunningTime="2026-09-10 23:20:41.760419419 +0000 UTC m=+1141.462052364"
Sep 10 23:20:42.426878 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:42.426837    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:42.432471 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:42.432443    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:42.746382 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:42.746306    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:42.747354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:42.747337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:20:43.564455 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:43.564421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:43.665081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:43.665050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:43.765435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:43.765398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:44.802321 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.802286    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"]
Sep 10 23:20:44.806253 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.806233    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:44.808277 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.808255    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-76145153-serving-cert\""
Sep 10 23:20:44.818233 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.818206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"]
Sep 10 23:20:44.832973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.832937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:44.930600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.926973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-metrics-certs\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:44.930600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.927056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ac90e-cf64-4a94-a92a-d437c3ad783c-openshift-service-ca\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:44.930600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.927095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-maas-api-tls\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:44.930600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.927123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvtk\" (UniqueName: \"kubernetes.io/projected/df9ac90e-cf64-4a94-a92a-d437c3ad783c-kube-api-access-zpvtk\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:44.940810 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:44.940769    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:45.028650 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.028617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-metrics-certs\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:45.028853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.028682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ac90e-cf64-4a94-a92a-d437c3ad783c-openshift-service-ca\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:45.028853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.028715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-maas-api-tls\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:45.028853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.028741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvtk\" (UniqueName: \"kubernetes.io/projected/df9ac90e-cf64-4a94-a92a-d437c3ad783c-kube-api-access-zpvtk\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:45.029022 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:20:45.028874    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-76145153-serving-cert: secret "maas-api-e2e-mt-76145153-serving-cert" not found
Sep 10 23:20:45.029022 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:20:45.028957    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-maas-api-tls podName:df9ac90e-cf64-4a94-a92a-d437c3ad783c nodeName:}" failed. No retries permitted until 2026-09-10 23:20:45.528934561 +0000 UTC m=+1145.230567497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-maas-api-tls") pod "maas-api-e2e-mt-76145153-68fdd74c59-plbp5" (UID: "df9ac90e-cf64-4a94-a92a-d437c3ad783c") : secret "maas-api-e2e-mt-76145153-serving-cert" not found
Sep 10 23:20:45.029555 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.029510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ac90e-cf64-4a94-a92a-d437c3ad783c-openshift-service-ca\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:45.031267 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.031236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-metrics-certs\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:45.036977 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.036957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvtk\" (UniqueName: \"kubernetes.io/projected/df9ac90e-cf64-4a94-a92a-d437c3ad783c-kube-api-access-zpvtk\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:45.106627 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.106591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:45.487653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.487528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:45.535111 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.535082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-maas-api-tls\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:45.537561 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.537523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-maas-api-tls\") pod \"maas-api-e2e-mt-76145153-68fdd74c59-plbp5\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:45.550204 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.550178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:45.721882 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:45.721844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:46.003889 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:46.003866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"]
Sep 10 23:20:46.005161 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:20:46.005130    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9ac90e_cf64_4a94_a92a_d437c3ad783c.slice/crio-cc9bec24ef5f1a566c32b556b0e439cfc050c64a4ba9bf5aa109a2272cf71152 WatchSource:0}: Error finding container cc9bec24ef5f1a566c32b556b0e439cfc050c64a4ba9bf5aa109a2272cf71152: Status 404 returned error can't find the container with id cc9bec24ef5f1a566c32b556b0e439cfc050c64a4ba9bf5aa109a2272cf71152
Sep 10 23:20:46.777897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:46.777814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5" event={"ID":"df9ac90e-cf64-4a94-a92a-d437c3ad783c","Type":"ContainerStarted","Data":"089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134"}
Sep 10 23:20:46.777897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:46.777862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5" event={"ID":"df9ac90e-cf64-4a94-a92a-d437c3ad783c","Type":"ContainerStarted","Data":"cc9bec24ef5f1a566c32b556b0e439cfc050c64a4ba9bf5aa109a2272cf71152"}
Sep 10 23:20:46.778077 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:46.777921    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:46.798664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:46.798615    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5" podStartSLOduration=2.343819444 podStartE2EDuration="2.798589949s" totalImagesPullingTime="454.770505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:46.79568378 +0000 UTC m=+1146.497316724" watchObservedRunningTime="2026-09-10 23:20:46.798589949 +0000 UTC m=+1146.500222892"
Sep 10 23:20:52.790697 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:52.790665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:20:53.586973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:53.586938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:54.146827 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:54.146786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:54.148983 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:54.148962    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6"]
Sep 10 23:20:54.159402 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:54.159376    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-679c94-29818035-gnzn6"]
Sep 10 23:20:54.233173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:54.233147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:54.333427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:54.333393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:54.387753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:54.387715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:54.696274 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:54.696246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:54.834425 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:54.834387    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250a7699-a794-4ddc-a08f-676ceea369a5" path="/var/lib/kubelet/pods/250a7699-a794-4ddc-a08f-676ceea369a5/volumes"
Sep 10 23:20:57.581720 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:57.581686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:20:59.684738 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:59.684701    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"]
Sep 10 23:20:59.793640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:20:59.793611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:03.151632 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:03.151421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:03.866385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:03.866347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:05.809477 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:05.809386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:08.503612 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:08.503580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:08.920237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:08.920207    2580 generic.go:356] "Generic (PLEG): container finished" podID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerID="a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce" exitCode=0
Sep 10 23:21:08.920488 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:08.920289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" event={"ID":"1673bcae-c356-4d22-9f5a-5ec8a515d22b","Type":"ContainerDied","Data":"a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce"}
Sep 10 23:21:08.920714 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:08.920694    2580 scope.go:117] "RemoveContainer" containerID="a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce"
Sep 10 23:21:08.921857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:08.921827    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:21:08.921947 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:08.921897    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:21:08.921947 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:08.921929    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:21:08.953038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:08.953008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:09.052696 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:09.052667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:09.935300 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:09.935252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" event={"ID":"1673bcae-c356-4d22-9f5a-5ec8a515d22b","Type":"ContainerStarted","Data":"363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6"}
Sep 10 23:21:10.426578 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:10.426522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:21:10.427996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:10.427959    2580 patch_prober.go:28] interesting pod/e2e-mt-76145153-openshift-default-979f65fcb-q7g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 23:21:10.428149 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:10.428053    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 23:21:11.427038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:11.426992    2580 patch_prober.go:28] interesting pod/e2e-mt-76145153-openshift-default-979f65fcb-q7g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 23:21:11.427573 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:11.427080    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 23:21:12.427325 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:12.427279    2580 patch_prober.go:28] interesting pod/e2e-mt-76145153-openshift-default-979f65fcb-q7g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 23:21:12.427843 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:12.427383    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 23:21:12.747376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:12.747288    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:21:13.426527 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:13.426494    2580 patch_prober.go:28] interesting pod/e2e-mt-76145153-openshift-default-979f65fcb-q7g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 23:21:13.426857 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:13.426825    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 23:21:14.426457 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.426421    2580 patch_prober.go:28] interesting pod/e2e-mt-76145153-openshift-default-979f65fcb-q7g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 23:21:14.427078 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.427044    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 23:21:14.705318 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.705226    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc" podUID="813110f8-3d6f-49ba-a1f3-169ae1fe69d8" containerName="main" containerID="cri-o://94ab949f248402ee693377634f70c20279d860aa92f96bafdcb090deef23cad9" gracePeriod=45
Sep 10 23:21:14.974702 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.974673    2580 generic.go:356] "Generic (PLEG): container finished" podID="813110f8-3d6f-49ba-a1f3-169ae1fe69d8" containerID="94ab949f248402ee693377634f70c20279d860aa92f96bafdcb090deef23cad9" exitCode=0
Sep 10 23:21:14.974861 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.974748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc" event={"ID":"813110f8-3d6f-49ba-a1f3-169ae1fe69d8","Type":"ContainerDied","Data":"94ab949f248402ee693377634f70c20279d860aa92f96bafdcb090deef23cad9"}
Sep 10 23:21:14.974861 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.974795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc" event={"ID":"813110f8-3d6f-49ba-a1f3-169ae1fe69d8","Type":"ContainerDied","Data":"b77b200a0f6b91c5fbfc9b2e2e636f4dace18a79430e2ae47eabc395c06d4faf"}
Sep 10 23:21:14.974861 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.974811    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b77b200a0f6b91c5fbfc9b2e2e636f4dace18a79430e2ae47eabc395c06d4faf"
Sep 10 23:21:14.987738 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.987718    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:21:14.998271 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.998245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-dshm\") pod \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") "
Sep 10 23:21:14.998404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.998311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tmp-dir\") pod \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") "
Sep 10 23:21:14.998404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.998348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrgrh\" (UniqueName: \"kubernetes.io/projected/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-kube-api-access-vrgrh\") pod \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") "
Sep 10 23:21:14.998404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.998378    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-home\") pod \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") "
Sep 10 23:21:14.998652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.998515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-model-cache\") pod \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") "
Sep 10 23:21:14.998652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.998585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tls-certs\") pod \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\" (UID: \"813110f8-3d6f-49ba-a1f3-169ae1fe69d8\") "
Sep 10 23:21:14.998652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.998629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "813110f8-3d6f-49ba-a1f3-169ae1fe69d8" (UID: "813110f8-3d6f-49ba-a1f3-169ae1fe69d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:14.998801 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.998727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-home" (OuterVolumeSpecName: "home") pod "813110f8-3d6f-49ba-a1f3-169ae1fe69d8" (UID: "813110f8-3d6f-49ba-a1f3-169ae1fe69d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:14.998851 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.998822    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "813110f8-3d6f-49ba-a1f3-169ae1fe69d8" (UID: "813110f8-3d6f-49ba-a1f3-169ae1fe69d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:14.999108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.999086    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-home\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:14.999175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.999113    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-model-cache\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:14.999175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:14.999129    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tmp-dir\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:15.000654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:15.000626    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-dshm" (OuterVolumeSpecName: "dshm") pod "813110f8-3d6f-49ba-a1f3-169ae1fe69d8" (UID: "813110f8-3d6f-49ba-a1f3-169ae1fe69d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:15.000853 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:15.000833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "813110f8-3d6f-49ba-a1f3-169ae1fe69d8" (UID: "813110f8-3d6f-49ba-a1f3-169ae1fe69d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:21:15.000991 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:15.000972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-kube-api-access-vrgrh" (OuterVolumeSpecName: "kube-api-access-vrgrh") pod "813110f8-3d6f-49ba-a1f3-169ae1fe69d8" (UID: "813110f8-3d6f-49ba-a1f3-169ae1fe69d8"). InnerVolumeSpecName "kube-api-access-vrgrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:15.100695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:15.100666    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-dshm\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:15.100695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:15.100691    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrgrh\" (UniqueName: \"kubernetes.io/projected/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-kube-api-access-vrgrh\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:15.100695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:15.100701    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813110f8-3d6f-49ba-a1f3-169ae1fe69d8-tls-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:15.431148 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:15.431121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:21:15.432156 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:15.432137    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:21:15.980976 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:15.980951    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"
Sep 10 23:21:16.002099 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:16.002070    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"]
Sep 10 23:21:16.007325 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:16.007300    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-679c94/test-model-679c94-kserve-648c786fb9-64gsc"]
Sep 10 23:21:16.835084 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:16.835049    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="813110f8-3d6f-49ba-a1f3-169ae1fe69d8" path="/var/lib/kubelet/pods/813110f8-3d6f-49ba-a1f3-169ae1fe69d8/volumes"
Sep 10 23:21:22.185851 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:22.185812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:22.410589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:22.410535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:27.182071 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:27.182036    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"]
Sep 10 23:21:27.264744 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:27.264681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:28.970301 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:28.970267    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:21:32.892922 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:32.892893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:32.898719 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:32.898693    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq"]
Sep 10 23:21:32.901669 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:32.901645    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce17e-29818035-zfrcq"]
Sep 10 23:21:32.961324 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:32.961272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:33.023809 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:33.023778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:33.091885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:33.091857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:33.145918 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:33.145836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:33.433908 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:33.433824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:34.834508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:34.834467    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce8a510-0a87-494d-8980-08c331f3736b" path="/var/lib/kubelet/pods/4ce8a510-0a87-494d-8980-08c331f3736b/volumes"
Sep 10 23:21:36.723980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:36.723943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:36.954569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:36.954510    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"]
Sep 10 23:21:36.954854 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:36.954800    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy" containerID="cri-o://44f3444b2915b91de7124a1303a795e35693d4c637212de0d236749c9fb202f4" gracePeriod=30
Sep 10 23:21:36.968194 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:36.968167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:37.390998 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:37.390966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:39.427827 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:39.427789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:39.499655 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:39.499624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:39.670771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:39.670736    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"]
Sep 10 23:21:39.670999 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:39.670978    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" containerID="cri-o://7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f" gracePeriod=30
Sep 10 23:21:39.700769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:39.700686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:40.953911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:40.953878    2580 scope.go:117] "RemoveContainer" containerID="1d9f5420e08521070ebb259773001e6c92701849d7a9ea3ccc3e065979b09ace"
Sep 10 23:21:40.963103 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:40.963085    2580 scope.go:117] "RemoveContainer" containerID="07e33d6fea479222d6dafd01a8ad8b6cac138e1dca70b90bff3b3987c4f843b7"
Sep 10 23:21:42.016255 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:21:42.016217    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5252afc0_120c_4b0e_8414_369c39abd59b.slice/crio-conmon-44f3444b2915b91de7124a1303a795e35693d4c637212de0d236749c9fb202f4.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 23:21:42.157061 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.157029    2580 generic.go:356] "Generic (PLEG): container finished" podID="5252afc0-120c-4b0e-8414-369c39abd59b" containerID="44f3444b2915b91de7124a1303a795e35693d4c637212de0d236749c9fb202f4" exitCode=0
Sep 10 23:21:42.157233 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.157101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" event={"ID":"5252afc0-120c-4b0e-8414-369c39abd59b","Type":"ContainerDied","Data":"44f3444b2915b91de7124a1303a795e35693d4c637212de0d236749c9fb202f4"}
Sep 10 23:21:42.157233 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.157176    2580 scope.go:117] "RemoveContainer" containerID="7478dfb69d4774b6480d01f1fb37e1ac6d25cd01b1230eab6195f602c18f01fa"
Sep 10 23:21:42.202745 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.202718    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh" podUID="e40ccec9-5a51-4fbe-b514-e12a1279f85a" containerName="main" containerID="cri-o://d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a" gracePeriod=45
Sep 10 23:21:42.203499 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.203482    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:21:42.392319 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392276    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5252afc0-120c-4b0e-8414-369c39abd59b-istio-podinfo\") pod \"5252afc0-120c-4b0e-8414-369c39abd59b\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") "
Sep 10 23:21:42.392484 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-istio-token\") pod \"5252afc0-120c-4b0e-8414-369c39abd59b\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") "
Sep 10 23:21:42.392484 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-data\") pod \"5252afc0-120c-4b0e-8414-369c39abd59b\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") "
Sep 10 23:21:42.392484 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392466    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-socket\") pod \"5252afc0-120c-4b0e-8414-369c39abd59b\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") "
Sep 10 23:21:42.392731 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392501    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-envoy\") pod \"5252afc0-120c-4b0e-8414-369c39abd59b\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") "
Sep 10 23:21:42.392731 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-certs\") pod \"5252afc0-120c-4b0e-8414-369c39abd59b\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") "
Sep 10 23:21:42.392731 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-credential-socket\") pod \"5252afc0-120c-4b0e-8414-369c39abd59b\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") "
Sep 10 23:21:42.392731 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5252afc0-120c-4b0e-8414-369c39abd59b-istiod-ca-cert\") pod \"5252afc0-120c-4b0e-8414-369c39abd59b\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") "
Sep 10 23:21:42.392731 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl5cw\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-kube-api-access-tl5cw\") pod \"5252afc0-120c-4b0e-8414-369c39abd59b\" (UID: \"5252afc0-120c-4b0e-8414-369c39abd59b\") "
Sep 10 23:21:42.393046 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392735    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-data" (OuterVolumeSpecName: "istio-data") pod "5252afc0-120c-4b0e-8414-369c39abd59b" (UID: "5252afc0-120c-4b0e-8414-369c39abd59b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.393046 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392763    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5252afc0-120c-4b0e-8414-369c39abd59b" (UID: "5252afc0-120c-4b0e-8414-369c39abd59b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.393046 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5252afc0-120c-4b0e-8414-369c39abd59b" (UID: "5252afc0-120c-4b0e-8414-369c39abd59b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.393046 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.392974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5252afc0-120c-4b0e-8414-369c39abd59b" (UID: "5252afc0-120c-4b0e-8414-369c39abd59b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.393261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.393139    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.393261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.393159    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.393261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.393175    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.393261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.393188    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.393462 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.393265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5252afc0-120c-4b0e-8414-369c39abd59b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5252afc0-120c-4b0e-8414-369c39abd59b" (UID: "5252afc0-120c-4b0e-8414-369c39abd59b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:42.395376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.395343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5252afc0-120c-4b0e-8414-369c39abd59b" (UID: "5252afc0-120c-4b0e-8414-369c39abd59b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.395376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.395351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-istio-token" (OuterVolumeSpecName: "istio-token") pod "5252afc0-120c-4b0e-8414-369c39abd59b" (UID: "5252afc0-120c-4b0e-8414-369c39abd59b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:42.395531 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.395406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5252afc0-120c-4b0e-8414-369c39abd59b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5252afc0-120c-4b0e-8414-369c39abd59b" (UID: "5252afc0-120c-4b0e-8414-369c39abd59b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:21:42.395710 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.395687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-kube-api-access-tl5cw" (OuterVolumeSpecName: "kube-api-access-tl5cw") pod "5252afc0-120c-4b0e-8414-369c39abd59b" (UID: "5252afc0-120c-4b0e-8414-369c39abd59b"). InnerVolumeSpecName "kube-api-access-tl5cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:42.433046 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.433025    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:21:42.494188 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.494155    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5252afc0-120c-4b0e-8414-369c39abd59b-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.494188 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.494185    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.494188 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.494193    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5252afc0-120c-4b0e-8414-369c39abd59b-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.494404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.494200    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5252afc0-120c-4b0e-8414-369c39abd59b-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.494404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.494208    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl5cw\" (UniqueName: \"kubernetes.io/projected/5252afc0-120c-4b0e-8414-369c39abd59b-kube-api-access-tl5cw\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.595208 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.595177    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-dshm\") pod \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") "
Sep 10 23:21:42.595377 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.595218    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-model-cache\") pod \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") "
Sep 10 23:21:42.595377 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.595242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-home\") pod \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") "
Sep 10 23:21:42.595377 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.595319    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tmp-dir\") pod \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") "
Sep 10 23:21:42.595377 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.595351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frhh7\" (UniqueName: \"kubernetes.io/projected/e40ccec9-5a51-4fbe-b514-e12a1279f85a-kube-api-access-frhh7\") pod \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") "
Sep 10 23:21:42.595652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.595500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e40ccec9-5a51-4fbe-b514-e12a1279f85a" (UID: "e40ccec9-5a51-4fbe-b514-e12a1279f85a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.595652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.595510    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tls-certs\") pod \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\" (UID: \"e40ccec9-5a51-4fbe-b514-e12a1279f85a\") "
Sep 10 23:21:42.595652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.595591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e40ccec9-5a51-4fbe-b514-e12a1279f85a" (UID: "e40ccec9-5a51-4fbe-b514-e12a1279f85a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.595652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.595606    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-home" (OuterVolumeSpecName: "home") pod "e40ccec9-5a51-4fbe-b514-e12a1279f85a" (UID: "e40ccec9-5a51-4fbe-b514-e12a1279f85a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.596113 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.596089    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-model-cache\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.596216 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.596116    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-home\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.596216 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.596129    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tmp-dir\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.597619 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.597599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-dshm" (OuterVolumeSpecName: "dshm") pod "e40ccec9-5a51-4fbe-b514-e12a1279f85a" (UID: "e40ccec9-5a51-4fbe-b514-e12a1279f85a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.597696 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.597630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e40ccec9-5a51-4fbe-b514-e12a1279f85a" (UID: "e40ccec9-5a51-4fbe-b514-e12a1279f85a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:21:42.597696 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.597651    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40ccec9-5a51-4fbe-b514-e12a1279f85a-kube-api-access-frhh7" (OuterVolumeSpecName: "kube-api-access-frhh7") pod "e40ccec9-5a51-4fbe-b514-e12a1279f85a" (UID: "e40ccec9-5a51-4fbe-b514-e12a1279f85a"). InnerVolumeSpecName "kube-api-access-frhh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:42.697564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.697511    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frhh7\" (UniqueName: \"kubernetes.io/projected/e40ccec9-5a51-4fbe-b514-e12a1279f85a-kube-api-access-frhh7\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.697564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.697563    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ccec9-5a51-4fbe-b514-e12a1279f85a-tls-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.697564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:42.697574    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ccec9-5a51-4fbe-b514-e12a1279f85a-dshm\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:43.164340 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.164310    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"
Sep 10 23:21:43.164803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.164346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf" event={"ID":"5252afc0-120c-4b0e-8414-369c39abd59b","Type":"ContainerDied","Data":"89c22770ccd14e96289ea7079e5ecfa992536966aabfd5f4b2b2df04d1d034f6"}
Sep 10 23:21:43.164803 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.164402    2580 scope.go:117] "RemoveContainer" containerID="44f3444b2915b91de7124a1303a795e35693d4c637212de0d236749c9fb202f4"
Sep 10 23:21:43.166028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.166006    2580 generic.go:356] "Generic (PLEG): container finished" podID="e40ccec9-5a51-4fbe-b514-e12a1279f85a" containerID="d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a" exitCode=0
Sep 10 23:21:43.166107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.166042    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"
Sep 10 23:21:43.166107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.166060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh" event={"ID":"e40ccec9-5a51-4fbe-b514-e12a1279f85a","Type":"ContainerDied","Data":"d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a"}
Sep 10 23:21:43.166107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.166085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh" event={"ID":"e40ccec9-5a51-4fbe-b514-e12a1279f85a","Type":"ContainerDied","Data":"4215c044e3e4208c561d82c4d6d55e8ee7103bda1a23e50d4076a74c62c10214"}
Sep 10 23:21:43.174487 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.174471    2580 scope.go:117] "RemoveContainer" containerID="d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a"
Sep 10 23:21:43.183242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.183212    2580 scope.go:117] "RemoveContainer" containerID="d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a"
Sep 10 23:21:43.183493 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:21:43.183477    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a\": container with ID starting with d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a not found: ID does not exist" containerID="d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a"
Sep 10 23:21:43.183591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.183499    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a"} err="failed to get container status \"d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a\": rpc error: code = NotFound desc = could not find container \"d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a\": container with ID starting with d8db94b3f681304769923a288f0f284611f2f9055643726994d6e7ffcfadc60a not found: ID does not exist"
Sep 10 23:21:43.185009 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.184992    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"]
Sep 10 23:21:43.188974 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.188955    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-679c94-openshift-default-586587fdb6-727nf"]
Sep 10 23:21:43.197285 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.197264    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"]
Sep 10 23:21:43.199629 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.199614    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/simulator-w0-6ce17e-kserve-5859878bd6-7jpdh"]
Sep 10 23:21:43.330317 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.330233    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"]
Sep 10 23:21:43.349814 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.349787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:43.647161 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.647125    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 23:21:43.647304 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:43.647194    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:21:44.837364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:44.837331    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" path="/var/lib/kubelet/pods/5252afc0-120c-4b0e-8414-369c39abd59b/volumes"
Sep 10 23:21:44.838017 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:44.837994    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e40ccec9-5a51-4fbe-b514-e12a1279f85a" path="/var/lib/kubelet/pods/e40ccec9-5a51-4fbe-b514-e12a1279f85a/volumes"
Sep 10 23:21:44.917338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:44.917312    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:21:45.021170 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021078    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twgfk\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-kube-api-access-twgfk\") pod \"47562c88-cde0-4b21-8198-0e98316ce5c9\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") "
Sep 10 23:21:45.021170 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47562c88-cde0-4b21-8198-0e98316ce5c9-istiod-ca-cert\") pod \"47562c88-cde0-4b21-8198-0e98316ce5c9\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") "
Sep 10 23:21:45.021170 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021166    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-certs\") pod \"47562c88-cde0-4b21-8198-0e98316ce5c9\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") "
Sep 10 23:21:45.021441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-socket\") pod \"47562c88-cde0-4b21-8198-0e98316ce5c9\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") "
Sep 10 23:21:45.021441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021199    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-data\") pod \"47562c88-cde0-4b21-8198-0e98316ce5c9\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") "
Sep 10 23:21:45.021441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-token\") pod \"47562c88-cde0-4b21-8198-0e98316ce5c9\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") "
Sep 10 23:21:45.021441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021349    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-credential-socket\") pod \"47562c88-cde0-4b21-8198-0e98316ce5c9\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") "
Sep 10 23:21:45.021441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021395    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-podinfo\") pod \"47562c88-cde0-4b21-8198-0e98316ce5c9\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") "
Sep 10 23:21:45.021441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-envoy\") pod \"47562c88-cde0-4b21-8198-0e98316ce5c9\" (UID: \"47562c88-cde0-4b21-8198-0e98316ce5c9\") "
Sep 10 23:21:45.021771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021520    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47562c88-cde0-4b21-8198-0e98316ce5c9" (UID: "47562c88-cde0-4b21-8198-0e98316ce5c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:45.021771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021536    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47562c88-cde0-4b21-8198-0e98316ce5c9" (UID: "47562c88-cde0-4b21-8198-0e98316ce5c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:45.021771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021668    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47562c88-cde0-4b21-8198-0e98316ce5c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47562c88-cde0-4b21-8198-0e98316ce5c9" (UID: "47562c88-cde0-4b21-8198-0e98316ce5c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:45.021936 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021778    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "47562c88-cde0-4b21-8198-0e98316ce5c9" (UID: "47562c88-cde0-4b21-8198-0e98316ce5c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:45.021986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021948    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47562c88-cde0-4b21-8198-0e98316ce5c9" (UID: "47562c88-cde0-4b21-8198-0e98316ce5c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:45.021986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021962    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47562c88-cde0-4b21-8198-0e98316ce5c9-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:45.021986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021981    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:45.022138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.021994    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:45.022138 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.022007    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:45.024447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.024409    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47562c88-cde0-4b21-8198-0e98316ce5c9" (UID: "47562c88-cde0-4b21-8198-0e98316ce5c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:21:45.024447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.024408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47562c88-cde0-4b21-8198-0e98316ce5c9" (UID: "47562c88-cde0-4b21-8198-0e98316ce5c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:45.024447 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.024439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "47562c88-cde0-4b21-8198-0e98316ce5c9" (UID: "47562c88-cde0-4b21-8198-0e98316ce5c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:45.024679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.024462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-kube-api-access-twgfk" (OuterVolumeSpecName: "kube-api-access-twgfk") pod "47562c88-cde0-4b21-8198-0e98316ce5c9" (UID: "47562c88-cde0-4b21-8198-0e98316ce5c9"). InnerVolumeSpecName "kube-api-access-twgfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:45.122756 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.122727    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:45.122756 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.122755    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:45.122927 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.122765    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:45.122927 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.122774    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47562c88-cde0-4b21-8198-0e98316ce5c9-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:45.122927 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.122782    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgfk\" (UniqueName: \"kubernetes.io/projected/47562c88-cde0-4b21-8198-0e98316ce5c9-kube-api-access-twgfk\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:45.182099 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.182068    2580 generic.go:356] "Generic (PLEG): container finished" podID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerID="7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f" exitCode=0
Sep 10 23:21:45.182262 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.182117    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"
Sep 10 23:21:45.182262 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.182152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" event={"ID":"47562c88-cde0-4b21-8198-0e98316ce5c9","Type":"ContainerDied","Data":"7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f"}
Sep 10 23:21:45.182262 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.182216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk" event={"ID":"47562c88-cde0-4b21-8198-0e98316ce5c9","Type":"ContainerDied","Data":"cc2f696a2f70bafed2a27a0206a3d83432c8402d4a8799b2517deb34b6f02654"}
Sep 10 23:21:45.182262 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.182245    2580 scope.go:117] "RemoveContainer" containerID="7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f"
Sep 10 23:21:45.193347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.193333    2580 scope.go:117] "RemoveContainer" containerID="35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1"
Sep 10 23:21:45.201937 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.201919    2580 scope.go:117] "RemoveContainer" containerID="7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f"
Sep 10 23:21:45.202235 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:21:45.202213    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f\": container with ID starting with 7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f not found: ID does not exist" containerID="7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f"
Sep 10 23:21:45.202328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.202246    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f"} err="failed to get container status \"7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f\": rpc error: code = NotFound desc = could not find container \"7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f\": container with ID starting with 7a2d658ea4da60d902ff2cb7265c684a4618e43ed690dbbc968391d180d2b57f not found: ID does not exist"
Sep 10 23:21:45.202328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.202267    2580 scope.go:117] "RemoveContainer" containerID="35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1"
Sep 10 23:21:45.202695 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:21:45.202615    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1\": container with ID starting with 35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1 not found: ID does not exist" containerID="35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1"
Sep 10 23:21:45.202785 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.202699    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1"} err="failed to get container status \"35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1\": rpc error: code = NotFound desc = could not find container \"35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1\": container with ID starting with 35110152b02645d2a74a606c2d46b035e5e2f49935bc80cc3a2226b9a25fd9c1 not found: ID does not exist"
Sep 10 23:21:45.205717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.205697    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"]
Sep 10 23:21:45.209975 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:45.209955    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce17e-openshift-default-848b659b94-72crk"]
Sep 10 23:21:46.834206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:46.834170    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" path="/var/lib/kubelet/pods/47562c88-cde0-4b21-8198-0e98316ce5c9/volumes"
Sep 10 23:21:47.773375 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:47.773343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:47.782368 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:47.782322    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb"]
Sep 10 23:21:47.785737 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:47.785715    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-568856-29818035-fzncb"]
Sep 10 23:21:47.838051 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:47.838015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:47.897942 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:47.897908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:47.967050 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:47.967016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:48.020116 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:48.020077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:48.314674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:48.314645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:48.835079 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:48.835047    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e31206-e26b-4e21-b80c-1dda84dfd7dd" path="/var/lib/kubelet/pods/e5e31206-e26b-4e21-b80c-1dda84dfd7dd/volumes"
Sep 10 23:21:51.439304 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:51.439277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:53.469191 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:53.469158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:21:53.470130 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:53.470102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:58.350374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.350335    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g" podUID="f1b13246-fbbb-497a-8e6f-c99c94e9093d" containerName="main" containerID="cri-o://de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c" gracePeriod=45
Sep 10 23:21:58.602009 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.601944    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:21:58.651054 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-model-cache\") pod \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") "
Sep 10 23:21:58.651250 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlvln\" (UniqueName: \"kubernetes.io/projected/f1b13246-fbbb-497a-8e6f-c99c94e9093d-kube-api-access-mlvln\") pod \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") "
Sep 10 23:21:58.651250 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651127    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-home\") pod \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") "
Sep 10 23:21:58.651250 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-dshm\") pod \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") "
Sep 10 23:21:58.651250 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tls-certs\") pod \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") "
Sep 10 23:21:58.651250 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tmp-dir\") pod \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\" (UID: \"f1b13246-fbbb-497a-8e6f-c99c94e9093d\") "
Sep 10 23:21:58.651518 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651307    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1b13246-fbbb-497a-8e6f-c99c94e9093d" (UID: "f1b13246-fbbb-497a-8e6f-c99c94e9093d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:58.651518 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-home" (OuterVolumeSpecName: "home") pod "f1b13246-fbbb-497a-8e6f-c99c94e9093d" (UID: "f1b13246-fbbb-497a-8e6f-c99c94e9093d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:58.651675 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651558    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1b13246-fbbb-497a-8e6f-c99c94e9093d" (UID: "f1b13246-fbbb-497a-8e6f-c99c94e9093d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:58.651723 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651679    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-model-cache\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:58.651723 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651694    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-home\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:58.651723 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.651721    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tmp-dir\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:58.653557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.653515    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-dshm" (OuterVolumeSpecName: "dshm") pod "f1b13246-fbbb-497a-8e6f-c99c94e9093d" (UID: "f1b13246-fbbb-497a-8e6f-c99c94e9093d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:58.653662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.653599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b13246-fbbb-497a-8e6f-c99c94e9093d-kube-api-access-mlvln" (OuterVolumeSpecName: "kube-api-access-mlvln") pod "f1b13246-fbbb-497a-8e6f-c99c94e9093d" (UID: "f1b13246-fbbb-497a-8e6f-c99c94e9093d"). InnerVolumeSpecName "kube-api-access-mlvln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:58.653662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.653612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1b13246-fbbb-497a-8e6f-c99c94e9093d" (UID: "f1b13246-fbbb-497a-8e6f-c99c94e9093d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:21:58.752602 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.752531    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b13246-fbbb-497a-8e6f-c99c94e9093d-tls-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:58.752602 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.752596    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvln\" (UniqueName: \"kubernetes.io/projected/f1b13246-fbbb-497a-8e6f-c99c94e9093d-kube-api-access-mlvln\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:58.752602 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:58.752605    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b13246-fbbb-497a-8e6f-c99c94e9093d-dshm\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:21:59.270367 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:59.270339    2580 generic.go:356] "Generic (PLEG): container finished" podID="f1b13246-fbbb-497a-8e6f-c99c94e9093d" containerID="de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c" exitCode=0
Sep 10 23:21:59.270563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:59.270378    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"
Sep 10 23:21:59.270563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:59.270434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g" event={"ID":"f1b13246-fbbb-497a-8e6f-c99c94e9093d","Type":"ContainerDied","Data":"de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c"}
Sep 10 23:21:59.270563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:59.270476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g" event={"ID":"f1b13246-fbbb-497a-8e6f-c99c94e9093d","Type":"ContainerDied","Data":"0f145f292ad58a7322a92da3677f85f7e7fa4a993a32fc590ebf7a91302a38a8"}
Sep 10 23:21:59.270563 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:59.270498    2580 scope.go:117] "RemoveContainer" containerID="de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c"
Sep 10 23:21:59.279810 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:59.279781    2580 scope.go:117] "RemoveContainer" containerID="de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c"
Sep 10 23:21:59.280077 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:21:59.280059    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c\": container with ID starting with de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c not found: ID does not exist" containerID="de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c"
Sep 10 23:21:59.280127 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:59.280086    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c"} err="failed to get container status \"de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c\": rpc error: code = NotFound desc = could not find container \"de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c\": container with ID starting with de1656c362fdb03a621562a438eeea010e39391fdad3597bb7b5053c7fbe9e6c not found: ID does not exist"
Sep 10 23:21:59.286069 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:59.286045    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"]
Sep 10 23:21:59.289770 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:21:59.289737    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-568856/test-model-568856-kserve-5964b7957c-wqc2g"]
Sep 10 23:22:00.510959 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:00.510923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:00.564562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:00.564503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:00.836151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:00.836071    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1b13246-fbbb-497a-8e6f-c99c94e9093d" path="/var/lib/kubelet/pods/f1b13246-fbbb-497a-8e6f-c99c94e9093d/volumes"
Sep 10 23:22:02.075481 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:02.075447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:04.477745 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:04.477704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:04.578295 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:04.578265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:04.679740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:04.679688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:13.442199 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:13.442167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:18.979632 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:18.979593    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:22:22.564960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:22.564917    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"]
Sep 10 23:22:22.565427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:22.565173    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq" podUID="15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" containerName="istio-proxy" containerID="cri-o://13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62" gracePeriod=30
Sep 10 23:22:22.583387 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:22.583333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:22.800320 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:22.800288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:22.900597 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:22.900561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:26.178727 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:26.178699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:27.821706 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.821675    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:22:27.871377 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871347    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzcxl\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-kube-api-access-fzcxl\") pod \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") "
Sep 10 23:22:27.871583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-credential-socket\") pod \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") "
Sep 10 23:22:27.871583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-podinfo\") pod \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") "
Sep 10 23:22:27.871583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-token\") pod \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") "
Sep 10 23:22:27.871583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-data\") pod \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") "
Sep 10 23:22:27.871801 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-socket\") pod \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") "
Sep 10 23:22:27.871801 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-envoy\") pod \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") "
Sep 10 23:22:27.871801 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871706    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-certs\") pod \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") "
Sep 10 23:22:27.871801 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istiod-ca-cert\") pod \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\" (UID: \"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100\") "
Sep 10 23:22:27.872090 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871810    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" (UID: "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:27.872090 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.871852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-data" (OuterVolumeSpecName: "istio-data") pod "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" (UID: "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:27.872090 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.872022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" (UID: "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:27.872251 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.872104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" (UID: "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:27.872251 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.872179    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" (UID: "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:22:27.872660 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.872589    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:27.872660 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.872614    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:27.872660 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.872629    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:27.872911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.872674    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:27.872911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.872779    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:27.874066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.874038    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-token" (OuterVolumeSpecName: "istio-token") pod "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" (UID: "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:22:27.874165 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.874097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" (UID: "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:22:27.874213 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.874173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-kube-api-access-fzcxl" (OuterVolumeSpecName: "kube-api-access-fzcxl") pod "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" (UID: "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100"). InnerVolumeSpecName "kube-api-access-fzcxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:22:27.874320 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.874302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" (UID: "15d779c6-ebd2-4487-b8f7-6f9f6aa2c100"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:27.926913 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.926858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:27.973989 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.973953    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:27.973989 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.973981    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzcxl\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-kube-api-access-fzcxl\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:27.973989 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.973991    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:27.973989 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:27.973999    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:28.452215 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.452183    2580 generic.go:356] "Generic (PLEG): container finished" podID="15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" containerID="13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62" exitCode=0
Sep 10 23:22:28.452401 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.452240    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"
Sep 10 23:22:28.452401 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.452277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq" event={"ID":"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100","Type":"ContainerDied","Data":"13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62"}
Sep 10 23:22:28.452401 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.452331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq" event={"ID":"15d779c6-ebd2-4487-b8f7-6f9f6aa2c100","Type":"ContainerDied","Data":"749d6eedea8f33ff29d0a9edcba67ea6ade28678104975914bfdfb38a9f27452"}
Sep 10 23:22:28.452401 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.452357    2580 scope.go:117] "RemoveContainer" containerID="13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62"
Sep 10 23:22:28.462640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.462621    2580 scope.go:117] "RemoveContainer" containerID="13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62"
Sep 10 23:22:28.462915 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:22:28.462897    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62\": container with ID starting with 13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62 not found: ID does not exist" containerID="13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62"
Sep 10 23:22:28.462956 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.462928    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62"} err="failed to get container status \"13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62\": rpc error: code = NotFound desc = could not find container \"13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62\": container with ID starting with 13a27c838c35c714c245b95ecbe4b4d74565d9c6a0d9989416252326cd165d62 not found: ID does not exist"
Sep 10 23:22:28.472702 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.472680    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"]
Sep 10 23:22:28.476524 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.476502    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-c93e93-openshift-default-86965d8768-cf2rq"]
Sep 10 23:22:28.836056 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.835976    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" path="/var/lib/kubelet/pods/15d779c6-ebd2-4487-b8f7-6f9f6aa2c100/volumes"
Sep 10 23:22:28.908708 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:28.908677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:30.388144 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:30.388103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:30.662636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:30.662556    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"]
Sep 10 23:22:30.662783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:30.662749    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy" containerID="cri-o://30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f" gracePeriod=30
Sep 10 23:22:30.676823 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:30.676798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:31.189630 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.189599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:31.466059 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.465967    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"]
Sep 10 23:22:31.466592 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466572    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" containerName="istio-proxy"
Sep 10 23:22:31.466592 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466594    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" containerName="istio-proxy"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466621    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466629    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466647    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466669    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466679    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466689    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e40ccec9-5a51-4fbe-b514-e12a1279f85a" containerName="main"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466699    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e40ccec9-5a51-4fbe-b514-e12a1279f85a" containerName="main"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466708    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813110f8-3d6f-49ba-a1f3-169ae1fe69d8" containerName="main"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466716    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="813110f8-3d6f-49ba-a1f3-169ae1fe69d8" containerName="main"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466728    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1b13246-fbbb-497a-8e6f-c99c94e9093d" containerName="main"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466735    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1b13246-fbbb-497a-8e6f-c99c94e9093d" containerName="main"
Sep 10 23:22:31.466742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466744    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy"
Sep 10 23:22:31.467386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466752    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy"
Sep 10 23:22:31.467386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466835    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15d779c6-ebd2-4487-b8f7-6f9f6aa2c100" containerName="istio-proxy"
Sep 10 23:22:31.467386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466846    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy"
Sep 10 23:22:31.467386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466855    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="813110f8-3d6f-49ba-a1f3-169ae1fe69d8" containerName="main"
Sep 10 23:22:31.467386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466867    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy"
Sep 10 23:22:31.467386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466875    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5252afc0-120c-4b0e-8414-369c39abd59b" containerName="istio-proxy"
Sep 10 23:22:31.467386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466885    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e40ccec9-5a51-4fbe-b514-e12a1279f85a" containerName="main"
Sep 10 23:22:31.467386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.466893    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1b13246-fbbb-497a-8e6f-c99c94e9093d" containerName="main"
Sep 10 23:22:31.470223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.470199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.472291 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.472256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6c97f2-openshift-default-dockercfg-bxxng\""
Sep 10 23:22:31.481728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.481706    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"]
Sep 10 23:22:31.492756 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.492732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:31.611424 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.611330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-podinfo\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.611637 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.611449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-token\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.611637 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.611529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-socket\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.611637 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.611612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-data\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.611837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.611655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-envoy\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.611837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.611692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8bgx\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-kube-api-access-m8bgx\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.611837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.611735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istiod-ca-cert\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.611837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.611766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-credential-socket\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.611837 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.611821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-certs\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-credential-socket\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-certs\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-podinfo\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-token\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-socket\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-data\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713739 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-envoy\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713739 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8bgx\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-kube-api-access-m8bgx\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713739 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istiod-ca-cert\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713739 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-credential-socket\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.713959 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-certs\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.714009 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.713948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-data\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.714289 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.714269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istiod-ca-cert\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.714373 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.714310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-socket\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.716113 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.716046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-envoy\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.716375 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.716354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-podinfo\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.721304 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.721279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8bgx\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-kube-api-access-m8bgx\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.721483 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.721463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-token\") pod \"e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") " pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.784363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.784328    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:31.945485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:31.945437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:32.064638 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:32.064602    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"]
Sep 10 23:22:32.065929 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:22:32.065899    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baf19b7_ab1d_46fd_8a29_91a219b45b11.slice/crio-e9ee3281461a53498ebd9b40e61b3acf1f67a0cd4a455808ec79b94415330321 WatchSource:0}: Error finding container e9ee3281461a53498ebd9b40e61b3acf1f67a0cd4a455808ec79b94415330321: Status 404 returned error can't find the container with id e9ee3281461a53498ebd9b40e61b3acf1f67a0cd4a455808ec79b94415330321
Sep 10 23:22:32.068607 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:32.068573    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:22:32.068705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:32.068653    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:22:32.068705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:32.068683    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:22:32.483720 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:32.483677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" event={"ID":"5baf19b7-ab1d-46fd-8a29-91a219b45b11","Type":"ContainerStarted","Data":"c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a"}
Sep 10 23:22:32.484087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:32.483728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" event={"ID":"5baf19b7-ab1d-46fd-8a29-91a219b45b11","Type":"ContainerStarted","Data":"e9ee3281461a53498ebd9b40e61b3acf1f67a0cd4a455808ec79b94415330321"}
Sep 10 23:22:32.503533 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:32.503484    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podStartSLOduration=1.5034694210000001 podStartE2EDuration="1.503469421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:22:32.500256565 +0000 UTC m=+1252.201889510" watchObservedRunningTime="2026-09-10 23:22:32.503469421 +0000 UTC m=+1252.205102363"
Sep 10 23:22:32.784945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:32.784857    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:33.790108 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:33.790073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:34.350605 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:34.350569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:34.450223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:34.450192    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:34.498921 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:34.498888    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:34.500208 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:34.500187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:35.651503 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.651465    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"]
Sep 10 23:22:35.658174 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.658138    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="47562c88-cde0-4b21-8198-0e98316ce5c9" containerName="istio-proxy"
Sep 10 23:22:35.668417 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.668387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.672485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.672455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:35.673149 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.673120    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"]
Sep 10 23:22:35.675632 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.675608    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6c97f2-serving-cert\""
Sep 10 23:22:35.759575 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.759125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.759575 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.759209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.759575 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.759258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abecd3-d577-4522-90fb-9f40b8a12eba-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.759575 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.759333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/15abecd3-d577-4522-90fb-9f40b8a12eba-kube-api-access-n472l\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.771561 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.771516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:35.860640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.860604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.860806 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.860662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abecd3-d577-4522-90fb-9f40b8a12eba-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.860806 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.860732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/15abecd3-d577-4522-90fb-9f40b8a12eba-kube-api-access-n472l\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.860806 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:22:35.860770    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-serving-cert: secret "maas-api-e2e-shared-a-w3-6c97f2-serving-cert" not found
Sep 10 23:22:35.861041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.860829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.861041 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:22:35.860886    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-maas-api-tls podName:15abecd3-d577-4522-90fb-9f40b8a12eba nodeName:}" failed. No retries permitted until 2026-09-10 23:22:36.360863483 +0000 UTC m=+1256.062496415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-maas-api-tls") pod "maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m" (UID: "15abecd3-d577-4522-90fb-9f40b8a12eba") : secret "maas-api-e2e-shared-a-w3-6c97f2-serving-cert" not found
Sep 10 23:22:35.861325 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.861307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abecd3-d577-4522-90fb-9f40b8a12eba-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.863322 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.863304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.870806 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.870782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/15abecd3-d577-4522-90fb-9f40b8a12eba-kube-api-access-n472l\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:35.993209 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:35.993112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:36.135225 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.135196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:36.226771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.226740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:36.350833 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.350806    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:22:36.367954 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.367914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:36.371707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.371675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:36.469102 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-envoy\") pod \"b74498af-163d-4e8c-b2ff-196463327948\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") "
Sep 10 23:22:36.469102 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74498af-163d-4e8c-b2ff-196463327948-istio-podinfo\") pod \"b74498af-163d-4e8c-b2ff-196463327948\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") "
Sep 10 23:22:36.469364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469134    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-certs\") pod \"b74498af-163d-4e8c-b2ff-196463327948\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") "
Sep 10 23:22:36.469364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469167    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-data\") pod \"b74498af-163d-4e8c-b2ff-196463327948\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") "
Sep 10 23:22:36.469364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469208    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzwgb\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-kube-api-access-jzwgb\") pod \"b74498af-163d-4e8c-b2ff-196463327948\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") "
Sep 10 23:22:36.469364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469225    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-socket\") pod \"b74498af-163d-4e8c-b2ff-196463327948\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") "
Sep 10 23:22:36.469364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469289    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-istio-token\") pod \"b74498af-163d-4e8c-b2ff-196463327948\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") "
Sep 10 23:22:36.469364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-credential-socket\") pod \"b74498af-163d-4e8c-b2ff-196463327948\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") "
Sep 10 23:22:36.469743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74498af-163d-4e8c-b2ff-196463327948-istiod-ca-cert\") pod \"b74498af-163d-4e8c-b2ff-196463327948\" (UID: \"b74498af-163d-4e8c-b2ff-196463327948\") "
Sep 10 23:22:36.469743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b74498af-163d-4e8c-b2ff-196463327948" (UID: "b74498af-163d-4e8c-b2ff-196463327948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:36.469743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469495    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b74498af-163d-4e8c-b2ff-196463327948" (UID: "b74498af-163d-4e8c-b2ff-196463327948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:36.469743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-data" (OuterVolumeSpecName: "istio-data") pod "b74498af-163d-4e8c-b2ff-196463327948" (UID: "b74498af-163d-4e8c-b2ff-196463327948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:36.469743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469680    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b74498af-163d-4e8c-b2ff-196463327948" (UID: "b74498af-163d-4e8c-b2ff-196463327948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:36.470182 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74498af-163d-4e8c-b2ff-196463327948-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b74498af-163d-4e8c-b2ff-196463327948" (UID: "b74498af-163d-4e8c-b2ff-196463327948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:22:36.470182 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469906    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:36.470182 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469927    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:36.470182 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469941    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:36.470182 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.469949    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:36.471461 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.471440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b74498af-163d-4e8c-b2ff-196463327948" (UID: "b74498af-163d-4e8c-b2ff-196463327948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:36.471595 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.471463    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b74498af-163d-4e8c-b2ff-196463327948-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b74498af-163d-4e8c-b2ff-196463327948" (UID: "b74498af-163d-4e8c-b2ff-196463327948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:22:36.472082 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.472056    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-istio-token" (OuterVolumeSpecName: "istio-token") pod "b74498af-163d-4e8c-b2ff-196463327948" (UID: "b74498af-163d-4e8c-b2ff-196463327948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:22:36.472082 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.472070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-kube-api-access-jzwgb" (OuterVolumeSpecName: "kube-api-access-jzwgb") pod "b74498af-163d-4e8c-b2ff-196463327948" (UID: "b74498af-163d-4e8c-b2ff-196463327948"). InnerVolumeSpecName "kube-api-access-jzwgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:22:36.514149 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.514119    2580 generic.go:356] "Generic (PLEG): container finished" podID="b74498af-163d-4e8c-b2ff-196463327948" containerID="30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f" exitCode=0
Sep 10 23:22:36.514291 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.514166    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"
Sep 10 23:22:36.514291 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.514171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" event={"ID":"b74498af-163d-4e8c-b2ff-196463327948","Type":"ContainerDied","Data":"30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f"}
Sep 10 23:22:36.514291 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.514225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws" event={"ID":"b74498af-163d-4e8c-b2ff-196463327948","Type":"ContainerDied","Data":"a06e92f23e4cd344dbc7a73080f3d5c9753f2528ef53d21dd6107fce81dd7ff3"}
Sep 10 23:22:36.514291 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.514249    2580 scope.go:117] "RemoveContainer" containerID="30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f"
Sep 10 23:22:36.525440 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.525417    2580 scope.go:117] "RemoveContainer" containerID="e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b"
Sep 10 23:22:36.535515 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.535499    2580 scope.go:117] "RemoveContainer" containerID="30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f"
Sep 10 23:22:36.535880 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:22:36.535863    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f\": container with ID starting with 30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f not found: ID does not exist" containerID="30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f"
Sep 10 23:22:36.535942 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.535887    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f"} err="failed to get container status \"30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f\": rpc error: code = NotFound desc = could not find container \"30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f\": container with ID starting with 30156937ca0f123f792f82ac89af1e01088990b5472917ccce2c037d5079588f not found: ID does not exist"
Sep 10 23:22:36.535942 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.535905    2580 scope.go:117] "RemoveContainer" containerID="e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b"
Sep 10 23:22:36.536181 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:22:36.536163    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b\": container with ID starting with e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b not found: ID does not exist" containerID="e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b"
Sep 10 23:22:36.536244 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.536185    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b"} err="failed to get container status \"e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b\": rpc error: code = NotFound desc = could not find container \"e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b\": container with ID starting with e57cb244e6c739283229572ce81b1adadaaf47a6c5b9a6ef4cd0a4719c20f61b not found: ID does not exist"
Sep 10 23:22:36.536385 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.536353    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"]
Sep 10 23:22:36.539688 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.539667    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-568856-openshift-default-5d7f66dcf6-2pwws"]
Sep 10 23:22:36.571030 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.570978    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzwgb\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-kube-api-access-jzwgb\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:36.571030 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.571003    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74498af-163d-4e8c-b2ff-196463327948-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:36.571030 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.571013    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74498af-163d-4e8c-b2ff-196463327948-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:36.571030 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.571021    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74498af-163d-4e8c-b2ff-196463327948-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:36.571030 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.571028    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74498af-163d-4e8c-b2ff-196463327948-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:22:36.641717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.641687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:36.836029 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.835949    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b74498af-163d-4e8c-b2ff-196463327948" path="/var/lib/kubelet/pods/b74498af-163d-4e8c-b2ff-196463327948/volumes"
Sep 10 23:22:36.916271 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:36.916243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"]
Sep 10 23:22:36.917709 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:22:36.917673    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15abecd3_d577_4522_90fb_9f40b8a12eba.slice/crio-9b0529f6e91e8e7bd2d143e06bacd5d3e296a108de6c16b451ba6b57fb5e6ad6 WatchSource:0}: Error finding container 9b0529f6e91e8e7bd2d143e06bacd5d3e296a108de6c16b451ba6b57fb5e6ad6: Status 404 returned error can't find the container with id 9b0529f6e91e8e7bd2d143e06bacd5d3e296a108de6c16b451ba6b57fb5e6ad6
Sep 10 23:22:37.523021 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:37.522987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m" event={"ID":"15abecd3-d577-4522-90fb-9f40b8a12eba","Type":"ContainerStarted","Data":"42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64"}
Sep 10 23:22:37.523193 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:37.523038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m" event={"ID":"15abecd3-d577-4522-90fb-9f40b8a12eba","Type":"ContainerStarted","Data":"9b0529f6e91e8e7bd2d143e06bacd5d3e296a108de6c16b451ba6b57fb5e6ad6"}
Sep 10 23:22:37.523193 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:37.523071    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:37.539972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:37.539921    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m" podStartSLOduration=2.140578913 podStartE2EDuration="2.539899102s" totalImagesPullingTime="399.320189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:22:37.537968174 +0000 UTC m=+1257.239601117" watchObservedRunningTime="2026-09-10 23:22:37.539899102 +0000 UTC m=+1257.241532045"
Sep 10 23:22:37.605567 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:37.605515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:40.576190 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:40.576155    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:22:40.991555 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:40.991512    2580 scope.go:117] "RemoveContainer" containerID="c2bf9252a80392e27b732f1a28deb21f2c259c6f096469c6c7f2fe88295fa16b"
Sep 10 23:22:41.000905 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:41.000759    2580 scope.go:117] "RemoveContainer" containerID="94ab949f248402ee693377634f70c20279d860aa92f96bafdcb090deef23cad9"
Sep 10 23:22:43.541303 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:43.541260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:22:50.298495 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:50.298457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:50.539732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:50.539687    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 23:22:50.539899 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:50.539760    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 23:22:51.610970 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:51.610937    2580 generic.go:356] "Generic (PLEG): container finished" podID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerID="c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a" exitCode=0
Sep 10 23:22:51.611377 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:51.611007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" event={"ID":"5baf19b7-ab1d-46fd-8a29-91a219b45b11","Type":"ContainerDied","Data":"c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a"}
Sep 10 23:22:51.611419 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:51.611400    2580 scope.go:117] "RemoveContainer" containerID="c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a"
Sep 10 23:22:51.612697 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:51.612670    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:22:51.612767 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:51.612743    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:22:51.612810 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:51.612775    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:22:52.621279 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:52.621233    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" event={"ID":"5baf19b7-ab1d-46fd-8a29-91a219b45b11","Type":"ContainerStarted","Data":"b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f"}
Sep 10 23:22:52.785078 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:52.785041    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:52.785262 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:52.785172    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 23:22:52.785262 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:52.785210    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 23:22:53.784698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:53.784666    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 23:22:53.785061 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:53.784716    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 23:22:54.785365 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:54.785329    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 23:22:54.785759 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:54.785390    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 23:22:55.785069 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:55.785036    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 23:22:55.785246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:55.785091    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 23:22:56.046312 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:56.046212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:56.785120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:56.785084    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 23:22:56.785306 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:56.785144    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 23:22:57.785032 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:57.784992    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 23:22:57.785451 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:57.785052    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 23:22:58.788710 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:58.788682    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:58.789071 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:58.788941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:58.789673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:58.789656    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:22:59.293097 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.293029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:59.611617 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.611584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:59.879600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.879493    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"]
Sep 10 23:22:59.880132 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.880114    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy"
Sep 10 23:22:59.880204 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.880134    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy"
Sep 10 23:22:59.880266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.880257    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy"
Sep 10 23:22:59.883912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.883891    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:22:59.885858 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.885834    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-83be66-openshift-default-dockercfg-q72bk\""
Sep 10 23:22:59.895111 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.895078    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"]
Sep 10 23:22:59.907404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:22:59.907376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:00.013640 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.013603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh8g8\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-kube-api-access-fh8g8\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.013921 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.013902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-credential-socket\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.014033 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.014021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-podinfo\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.014156 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.014144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-token\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.014272 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.014258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-envoy\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.014369 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.014357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-socket\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.014492 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.014480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-data\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.014665 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.014648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99dc4886-7b32-41a8-b83d-28ea1fa90740-istiod-ca-cert\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.014800 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.014787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-certs\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.043634 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.043604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:00.116151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-data\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99dc4886-7b32-41a8-b83d-28ea1fa90740-istiod-ca-cert\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-certs\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8g8\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-kube-api-access-fh8g8\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-credential-socket\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-podinfo\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-token\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-envoy\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-socket\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-data\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-certs\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116886 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-credential-socket\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.116886 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.116797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-socket\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.117034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.117012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99dc4886-7b32-41a8-b83d-28ea1fa90740-istiod-ca-cert\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.119158 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.119134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-envoy\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.119276 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.119190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-podinfo\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.124005 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.123982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8g8\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-kube-api-access-fh8g8\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.124294 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.124272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-token\") pod \"e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") " pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.196293 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.196214    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:00.473494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.473469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"]
Sep 10 23:23:00.474599 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:23:00.474571    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99dc4886_7b32_41a8_b83d_28ea1fa90740.slice/crio-ee802e1b985dd4006a3b820c60855ae97bc6e79b5b4bbd62b77abee1db1fc4d1 WatchSource:0}: Error finding container ee802e1b985dd4006a3b820c60855ae97bc6e79b5b4bbd62b77abee1db1fc4d1: Status 404 returned error can't find the container with id ee802e1b985dd4006a3b820c60855ae97bc6e79b5b4bbd62b77abee1db1fc4d1
Sep 10 23:23:00.477095 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.477062    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:23:00.477211 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.477135    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:23:00.477211 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.477177    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:23:00.681424 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.681394    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" event={"ID":"99dc4886-7b32-41a8-b83d-28ea1fa90740","Type":"ContainerStarted","Data":"d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d"}
Sep 10 23:23:00.681648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.681451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" event={"ID":"99dc4886-7b32-41a8-b83d-28ea1fa90740","Type":"ContainerStarted","Data":"ee802e1b985dd4006a3b820c60855ae97bc6e79b5b4bbd62b77abee1db1fc4d1"}
Sep 10 23:23:00.701611 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:00.701557    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" podStartSLOduration=1.701521905 podStartE2EDuration="1.701521905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:23:00.697328717 +0000 UTC m=+1280.398961662" watchObservedRunningTime="2026-09-10 23:23:00.701521905 +0000 UTC m=+1280.403154849"
Sep 10 23:23:01.196695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:01.196665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:01.201807 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:01.201787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:01.687435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:01.687393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:01.688426 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:01.688410    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:02.570332 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:02.570302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:02.670929 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:02.670892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:03.693617 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.693584    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"]
Sep 10 23:23:03.694373 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.694350    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy"
Sep 10 23:23:03.694373 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.694376    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy"
Sep 10 23:23:03.694592 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.694490    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74498af-163d-4e8c-b2ff-196463327948" containerName="istio-proxy"
Sep 10 23:23:03.697969 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.697943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.702180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.702156    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-83be66-serving-cert\""
Sep 10 23:23:03.723287 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.723219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"]
Sep 10 23:23:03.731636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.731588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:03.755970 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.755929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.756178 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.756150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.756302 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.756248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c0c3a5-72e2-4cbe-b293-7c59e125e001-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.756355 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.756298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t28rv\" (UniqueName: \"kubernetes.io/projected/28c0c3a5-72e2-4cbe-b293-7c59e125e001-kube-api-access-t28rv\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.830514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.830484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:03.857836 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.857804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.858020 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.857927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.858020 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.857978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c0c3a5-72e2-4cbe-b293-7c59e125e001-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.858020 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.858007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t28rv\" (UniqueName: \"kubernetes.io/projected/28c0c3a5-72e2-4cbe-b293-7c59e125e001-kube-api-access-t28rv\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.858181 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:23:03.858090    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-serving-cert: secret "maas-api-e2e-shared-b-w3-83be66-serving-cert" not found
Sep 10 23:23:03.858234 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:23:03.858207    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-maas-api-tls podName:28c0c3a5-72e2-4cbe-b293-7c59e125e001 nodeName:}" failed. No retries permitted until 2026-09-10 23:23:04.358183952 +0000 UTC m=+1284.059816890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-maas-api-tls") pod "maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl" (UID: "28c0c3a5-72e2-4cbe-b293-7c59e125e001") : secret "maas-api-e2e-shared-b-w3-83be66-serving-cert" not found
Sep 10 23:23:03.858668 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.858648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c0c3a5-72e2-4cbe-b293-7c59e125e001-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.860530 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.860508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:03.867885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:03.867860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t28rv\" (UniqueName: \"kubernetes.io/projected/28c0c3a5-72e2-4cbe-b293-7c59e125e001-kube-api-access-t28rv\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:04.205102 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:04.205063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:04.299933 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:04.299896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:04.363642 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:04.363601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:04.366391 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:04.366356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:04.366725 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:04.366709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:04.628001 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:04.627970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:04.901397 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:04.901359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"]
Sep 10 23:23:04.902407 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:23:04.902384    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c0c3a5_72e2_4cbe_b293_7c59e125e001.slice/crio-6668bedb2eee0028d26a910d046d79dcf9952b97ed4af0f6fd5a9a59e0bb2505 WatchSource:0}: Error finding container 6668bedb2eee0028d26a910d046d79dcf9952b97ed4af0f6fd5a9a59e0bb2505: Status 404 returned error can't find the container with id 6668bedb2eee0028d26a910d046d79dcf9952b97ed4af0f6fd5a9a59e0bb2505
Sep 10 23:23:05.723458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:05.723356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl" event={"ID":"28c0c3a5-72e2-4cbe-b293-7c59e125e001","Type":"ContainerStarted","Data":"ecfaa726f483763f4c7e726c6a2d57b657567f720035e510520625d36176675a"}
Sep 10 23:23:05.723458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:05.723424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl" event={"ID":"28c0c3a5-72e2-4cbe-b293-7c59e125e001","Type":"ContainerStarted","Data":"6668bedb2eee0028d26a910d046d79dcf9952b97ed4af0f6fd5a9a59e0bb2505"}
Sep 10 23:23:05.724365 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:05.724335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:05.743393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:05.743345    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl" podStartSLOduration=2.208195868 podStartE2EDuration="2.743329191s" totalImagesPullingTime="535.133323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:23:05.740510956 +0000 UTC m=+1285.442143897" watchObservedRunningTime="2026-09-10 23:23:05.743329191 +0000 UTC m=+1285.444962134"
Sep 10 23:23:12.741378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:12.741350    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:23:19.378334 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:19.378301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:23:27.521976 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:27.521936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:29.880747 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:29.880717    2580 generic.go:356] "Generic (PLEG): container finished" podID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerID="d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d" exitCode=0
Sep 10 23:23:29.881137 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:29.880788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" event={"ID":"99dc4886-7b32-41a8-b83d-28ea1fa90740","Type":"ContainerDied","Data":"d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d"}
Sep 10 23:23:29.881235 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:29.881221    2580 scope.go:117] "RemoveContainer" containerID="d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d"
Sep 10 23:23:29.882441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:29.882408    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:23:29.882508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:29.882478    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:23:29.882556 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:29.882511    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:23:30.888747 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:30.888717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" event={"ID":"99dc4886-7b32-41a8-b83d-28ea1fa90740","Type":"ContainerStarted","Data":"2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f"}
Sep 10 23:23:31.196927 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:31.196842    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:31.197884 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:31.197859    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 23:23:31.198005 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:31.197907    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 23:23:31.688063 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:31.688027    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:32.197125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:32.197090    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 23:23:32.197500 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:32.197147    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 23:23:33.197430 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:33.197394    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 23:23:33.197845 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:33.197463    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 23:23:34.196936 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:34.196903    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 23:23:34.197111 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:34.196965    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 23:23:35.196826 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:35.196786    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 10 23:23:35.197203 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:35.196847    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 10 23:23:36.081754 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:36.081716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:36.201025 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:36.200999    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:36.202054 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:36.202033    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:23:36.306771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:36.306738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:42.979644 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:42.979616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:23:51.709472 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:51.709437    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"]
Sep 10 23:23:51.709909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:51.709674    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5" podUID="df9ac90e-cf64-4a94-a92a-d437c3ad783c" containerName="maas-api" containerID="cri-o://089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134" gracePeriod=30
Sep 10 23:23:51.730237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:51.730207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:51.827233 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:51.827205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:51.908257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:51.908220    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:51.960758 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:51.960693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:51.967491 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:51.967473    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:23:52.027566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.027258    2580 generic.go:356] "Generic (PLEG): container finished" podID="df9ac90e-cf64-4a94-a92a-d437c3ad783c" containerID="089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134" exitCode=0
Sep 10 23:23:52.027566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.027396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5" event={"ID":"df9ac90e-cf64-4a94-a92a-d437c3ad783c","Type":"ContainerDied","Data":"089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134"}
Sep 10 23:23:52.027566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.027456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5" event={"ID":"df9ac90e-cf64-4a94-a92a-d437c3ad783c","Type":"ContainerDied","Data":"cc9bec24ef5f1a566c32b556b0e439cfc050c64a4ba9bf5aa109a2272cf71152"}
Sep 10 23:23:52.027566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.027488    2580 scope.go:117] "RemoveContainer" containerID="089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134"
Sep 10 23:23:52.027926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.027696    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"
Sep 10 23:23:52.045240 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.045214    2580 scope.go:117] "RemoveContainer" containerID="089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134"
Sep 10 23:23:52.045849 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:23:52.045820    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134\": container with ID starting with 089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134 not found: ID does not exist" containerID="089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134"
Sep 10 23:23:52.045940 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.045863    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134"} err="failed to get container status \"089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134\": rpc error: code = NotFound desc = could not find container \"089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134\": container with ID starting with 089c7232e4562db9eb57334cb618fd039796f9383461dc159fb11d561f809134 not found: ID does not exist"
Sep 10 23:23:52.067211 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.067181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-metrics-certs\") pod \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") "
Sep 10 23:23:52.067392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.067293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpvtk\" (UniqueName: \"kubernetes.io/projected/df9ac90e-cf64-4a94-a92a-d437c3ad783c-kube-api-access-zpvtk\") pod \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") "
Sep 10 23:23:52.067392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.067360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-maas-api-tls\") pod \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") "
Sep 10 23:23:52.067515 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.067407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ac90e-cf64-4a94-a92a-d437c3ad783c-openshift-service-ca\") pod \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\" (UID: \"df9ac90e-cf64-4a94-a92a-d437c3ad783c\") "
Sep 10 23:23:52.067806 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.067772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9ac90e-cf64-4a94-a92a-d437c3ad783c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df9ac90e-cf64-4a94-a92a-d437c3ad783c" (UID: "df9ac90e-cf64-4a94-a92a-d437c3ad783c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:23:52.069411 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.069387    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df9ac90e-cf64-4a94-a92a-d437c3ad783c" (UID: "df9ac90e-cf64-4a94-a92a-d437c3ad783c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:23:52.069597 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.069577    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9ac90e-cf64-4a94-a92a-d437c3ad783c-kube-api-access-zpvtk" (OuterVolumeSpecName: "kube-api-access-zpvtk") pod "df9ac90e-cf64-4a94-a92a-d437c3ad783c" (UID: "df9ac90e-cf64-4a94-a92a-d437c3ad783c"). InnerVolumeSpecName "kube-api-access-zpvtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:23:52.069597 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.069590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df9ac90e-cf64-4a94-a92a-d437c3ad783c" (UID: "df9ac90e-cf64-4a94-a92a-d437c3ad783c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:23:52.168443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.168408    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpvtk\" (UniqueName: \"kubernetes.io/projected/df9ac90e-cf64-4a94-a92a-d437c3ad783c-kube-api-access-zpvtk\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:23:52.168443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.168434    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:23:52.168443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.168444    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ac90e-cf64-4a94-a92a-d437c3ad783c-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:23:52.168704 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.168454    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9ac90e-cf64-4a94-a92a-d437c3ad783c-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:23:52.350025 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.349990    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"]
Sep 10 23:23:52.352528 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.352507    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76145153-68fdd74c59-plbp5"]
Sep 10 23:23:52.834811 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:52.834777    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9ac90e-cf64-4a94-a92a-d437c3ad783c" path="/var/lib/kubelet/pods/df9ac90e-cf64-4a94-a92a-d437c3ad783c/volumes"
Sep 10 23:23:54.939616 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:54.939584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:59.959445 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:23:59.959410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:06.570331 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:06.570254    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:24:08.019044 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:08.019011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:10.059403 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:10.059372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:12.287749 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:12.287714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:16.436772 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:16.436734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:16.537553 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:16.537518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:19.973280 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:19.973200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:24:34.593276 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:34.593244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:36.701683 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:36.701654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:37.144035 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:37.144004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:42.787880 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:42.787845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:46.586163 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:46.586127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:50.779465 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:50.779432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:53.674185 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:53.674149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:24:57.494863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:57.494822    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:57.594659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:57.594631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:57.728816 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:57.728785    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"]
Sep 10 23:24:57.729104 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:57.729080    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy" containerID="cri-o://363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6" gracePeriod=30
Sep 10 23:24:57.769587 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:57.769460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:58.971992 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:58.971951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:59.074158 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:59.074124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:59.175012 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:59.174979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:59.748637 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:59.748607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:59.842607 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:59.842574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:59.943753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:24:59.943718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:00.434834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:00.434800    2580 patch_prober.go:28] interesting pod/e2e-mt-76145153-openshift-default-979f65fcb-q7g24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 23:25:00.435204 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:00.434869    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:25:00.724277 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:00.724184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:01.155677 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:01.155645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:02.988678 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:02.988655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:25:03.071782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.071688    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-socket\") pod \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") "
Sep 10 23:25:03.071782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.071734    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-envoy\") pod \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") "
Sep 10 23:25:03.071782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.071768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-token\") pod \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") "
Sep 10 23:25:03.072066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.071817    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-credential-socket\") pod \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") "
Sep 10 23:25:03.072066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.071879    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istiod-ca-cert\") pod \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") "
Sep 10 23:25:03.072066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.071921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-podinfo\") pod \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") "
Sep 10 23:25:03.072066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.071952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-data\") pod \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") "
Sep 10 23:25:03.072066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.071986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1673bcae-c356-4d22-9f5a-5ec8a515d22b" (UID: "1673bcae-c356-4d22-9f5a-5ec8a515d22b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:25:03.072066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-certs\") pod \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") "
Sep 10 23:25:03.072066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlpn6\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-kube-api-access-qlpn6\") pod \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\" (UID: \"1673bcae-c356-4d22-9f5a-5ec8a515d22b\") "
Sep 10 23:25:03.072383 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1673bcae-c356-4d22-9f5a-5ec8a515d22b" (UID: "1673bcae-c356-4d22-9f5a-5ec8a515d22b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:25:03.072383 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072213    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-data" (OuterVolumeSpecName: "istio-data") pod "1673bcae-c356-4d22-9f5a-5ec8a515d22b" (UID: "1673bcae-c356-4d22-9f5a-5ec8a515d22b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:25:03.072383 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1673bcae-c356-4d22-9f5a-5ec8a515d22b" (UID: "1673bcae-c356-4d22-9f5a-5ec8a515d22b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:25:03.072510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072449    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:25:03.072510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072467    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:25:03.072510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072481    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:25:03.072510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072493    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:25:03.072708 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.072624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1673bcae-c356-4d22-9f5a-5ec8a515d22b" (UID: "1673bcae-c356-4d22-9f5a-5ec8a515d22b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:25:03.074049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.074028    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1673bcae-c356-4d22-9f5a-5ec8a515d22b" (UID: "1673bcae-c356-4d22-9f5a-5ec8a515d22b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:25:03.074485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.074465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1673bcae-c356-4d22-9f5a-5ec8a515d22b" (UID: "1673bcae-c356-4d22-9f5a-5ec8a515d22b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:25:03.074733 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.074715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-token" (OuterVolumeSpecName: "istio-token") pod "1673bcae-c356-4d22-9f5a-5ec8a515d22b" (UID: "1673bcae-c356-4d22-9f5a-5ec8a515d22b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:25:03.074797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.074756    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-kube-api-access-qlpn6" (OuterVolumeSpecName: "kube-api-access-qlpn6") pod "1673bcae-c356-4d22-9f5a-5ec8a515d22b" (UID: "1673bcae-c356-4d22-9f5a-5ec8a515d22b"). InnerVolumeSpecName "kube-api-access-qlpn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:25:03.173805 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.173764    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:25:03.173805 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.173797    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:25:03.173805 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.173810    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlpn6\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-kube-api-access-qlpn6\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:25:03.174035 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.173822    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:25:03.174035 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.173834    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1673bcae-c356-4d22-9f5a-5ec8a515d22b-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:25:03.465225 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.465188    2580 generic.go:356] "Generic (PLEG): container finished" podID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerID="363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6" exitCode=0
Sep 10 23:25:03.465398 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.465262    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"
Sep 10 23:25:03.465398 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.465276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" event={"ID":"1673bcae-c356-4d22-9f5a-5ec8a515d22b","Type":"ContainerDied","Data":"363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6"}
Sep 10 23:25:03.465398 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.465357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24" event={"ID":"1673bcae-c356-4d22-9f5a-5ec8a515d22b","Type":"ContainerDied","Data":"710d8894dc667ead774a80d79096b17fab2a0717cfec656077e5d417a1379160"}
Sep 10 23:25:03.465398 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.465380    2580 scope.go:117] "RemoveContainer" containerID="363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6"
Sep 10 23:25:03.475807 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.475789    2580 scope.go:117] "RemoveContainer" containerID="a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce"
Sep 10 23:25:03.484392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.484375    2580 scope.go:117] "RemoveContainer" containerID="363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6"
Sep 10 23:25:03.484706 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:25:03.484689    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6\": container with ID starting with 363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6 not found: ID does not exist" containerID="363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6"
Sep 10 23:25:03.484776 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.484714    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6"} err="failed to get container status \"363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6\": rpc error: code = NotFound desc = could not find container \"363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6\": container with ID starting with 363abb96118f82147e44c4bf777661db595b94d3b2deff5701786ba422a4b0e6 not found: ID does not exist"
Sep 10 23:25:03.484776 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.484730    2580 scope.go:117] "RemoveContainer" containerID="a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce"
Sep 10 23:25:03.484957 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:25:03.484943    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce\": container with ID starting with a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce not found: ID does not exist" containerID="a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce"
Sep 10 23:25:03.484994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.484960    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce"} err="failed to get container status \"a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce\": rpc error: code = NotFound desc = could not find container \"a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce\": container with ID starting with a37d197e56740553004a6c9e8d7cb0ce1ac658c2e53766b06b430b3ef0c804ce not found: ID does not exist"
Sep 10 23:25:03.488841 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.488818    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"]
Sep 10 23:25:03.492098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:03.492077    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76145153-openshift-default-979f65fcb-q7g24"]
Sep 10 23:25:04.834718 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:04.834681    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" path="/var/lib/kubelet/pods/1673bcae-c356-4d22-9f5a-5ec8a515d22b/volumes"
Sep 10 23:25:07.743461 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:07.743434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:07.803456 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:07.803423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:08.388292 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:08.388258    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:14.426453 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:14.426408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:14.530259 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:14.530228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:14.628067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:14.628028    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:24.066744 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:24.066699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:25:25.083414 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:25.083376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:25.302476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:25.302422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:27.857965 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:27.857924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:34.954353 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954280    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"]
Sep 10 23:25:34.954789 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954772    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy"
Sep 10 23:25:34.954832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954790    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy"
Sep 10 23:25:34.954832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954802    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy"
Sep 10 23:25:34.954832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954808    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy"
Sep 10 23:25:34.954832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954824    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9ac90e-cf64-4a94-a92a-d437c3ad783c" containerName="maas-api"
Sep 10 23:25:34.954832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954830    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9ac90e-cf64-4a94-a92a-d437c3ad783c" containerName="maas-api"
Sep 10 23:25:34.954984 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954942    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy"
Sep 10 23:25:34.954984 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954956    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1673bcae-c356-4d22-9f5a-5ec8a515d22b" containerName="istio-proxy"
Sep 10 23:25:34.954984 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.954974    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9ac90e-cf64-4a94-a92a-d437c3ad783c" containerName="maas-api"
Sep 10 23:25:34.958120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.958102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:34.962494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.962466    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"openshift-service-ca.crt\""
Sep 10 23:25:34.962494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.962479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"default-dockercfg-2lgbw\""
Sep 10 23:25:34.962717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.962512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"kube-root-ca.crt\""
Sep 10 23:25:34.962717 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.962519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"e2e-conflict-a-d74b488b-kserve-self-signed-certs\""
Sep 10 23:25:34.971840 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.971805    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"]
Sep 10 23:25:34.999901 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.999870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-dshm\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.000107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.999951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j658r\" (UniqueName: \"kubernetes.io/projected/f6fa7cc3-d57c-409e-94a5-89379502205b-kube-api-access-j658r\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.000107 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:34.999991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-home\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.000230 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.000112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-model-cache\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.000230 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.000142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fa7cc3-d57c-409e-94a5-89379502205b-tls-certs\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.000230 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.000180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-tmp-dir\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.000608 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.000588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:35.101493 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.101463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-dshm\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.101732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.101517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j658r\" (UniqueName: \"kubernetes.io/projected/f6fa7cc3-d57c-409e-94a5-89379502205b-kube-api-access-j658r\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.101732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.101551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-home\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.101732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.101595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-model-cache\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.101732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.101625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fa7cc3-d57c-409e-94a5-89379502205b-tls-certs\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.101732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.101662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-tmp-dir\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.102019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.102007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-home\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.102079 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.102022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-tmp-dir\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.102302 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.102281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-model-cache\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.104048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.104016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-dshm\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.104335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.104318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fa7cc3-d57c-409e-94a5-89379502205b-tls-certs\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.110300 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.110275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j658r\" (UniqueName: \"kubernetes.io/projected/f6fa7cc3-d57c-409e-94a5-89379502205b-kube-api-access-j658r\") pod \"e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.268132 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.268045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:35.550596 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.546415    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"]
Sep 10 23:25:35.550881 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:25:35.550679    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6fa7cc3_d57c_409e_94a5_89379502205b.slice/crio-1fb678cca5ef2ed0f59443a06bb2fb543245712d7c94947c01a22b1b09f9f431 WatchSource:0}: Error finding container 1fb678cca5ef2ed0f59443a06bb2fb543245712d7c94947c01a22b1b09f9f431: Status 404 returned error can't find the container with id 1fb678cca5ef2ed0f59443a06bb2fb543245712d7c94947c01a22b1b09f9f431
Sep 10 23:25:35.666321 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.666285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525" event={"ID":"f6fa7cc3-d57c-409e-94a5-89379502205b","Type":"ContainerStarted","Data":"7d8bb3a1d1af1de868b61b6d8701c8e7797e4362f0982de77647239e737228a3"}
Sep 10 23:25:35.666321 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.666328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525" event={"ID":"f6fa7cc3-d57c-409e-94a5-89379502205b","Type":"ContainerStarted","Data":"1fb678cca5ef2ed0f59443a06bb2fb543245712d7c94947c01a22b1b09f9f431"}
Sep 10 23:25:35.684202 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:35.684147    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525" podStartSLOduration=1.684131577 podStartE2EDuration="1.684131577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:25:35.682722776 +0000 UTC m=+1435.384355720" watchObservedRunningTime="2026-09-10 23:25:35.684131577 +0000 UTC m=+1435.385764510"
Sep 10 23:25:41.075958 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:41.075926    2580 scope.go:117] "RemoveContainer" containerID="51b96c1ba4bf1a3e50c23bdedeed692b8d144bc2fd8881120d92c9477f937ae1"
Sep 10 23:25:45.071272 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:45.071232    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:25:45.268583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:45.268527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:45.268754 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:45.268594    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:45.281620 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:45.281579    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:45.736718 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:45.736690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:25:46.207345 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:46.207311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:52.131152 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.131116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:52.196065 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.196031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:52.458231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.458134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:52.769772 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.769682    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"]
Sep 10 23:25:52.774904 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.774888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.777886 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.777863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a4c77c-openshift-default-dockercfg-txnp4\""
Sep 10 23:25:52.781516 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.781494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:52.791295 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.791269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"]
Sep 10 23:25:52.883473 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.883421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-data\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.883782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.883485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-certs\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.883782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.883530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-envoy\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.883782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.883613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-credential-socket\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.883782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.883637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eede480-ca51-42f6-8c7b-a21769018b92-istio-podinfo\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.883782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.883684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eede480-ca51-42f6-8c7b-a21769018b92-istiod-ca-cert\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.883782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.883707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-istio-token\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.884110 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.883952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-socket\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.884110 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.883996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5kgg\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-kube-api-access-v5kgg\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.984653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.984607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-socket\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.984653 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.984661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5kgg\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-kube-api-access-v5kgg\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.984911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.984721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-data\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.984911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.984751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-certs\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.984911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.984784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-envoy\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.984911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.984822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-credential-socket\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.984911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.984846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eede480-ca51-42f6-8c7b-a21769018b92-istio-podinfo\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.984911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.984911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eede480-ca51-42f6-8c7b-a21769018b92-istiod-ca-cert\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.985223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.984939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-istio-token\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.985223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.985090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-socket\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.985223 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.985136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-certs\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.985344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.985224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-credential-socket\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.985386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.985350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-data\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.985720 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.985695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eede480-ca51-42f6-8c7b-a21769018b92-istiod-ca-cert\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.987253 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.987232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-envoy\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.987668 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.987648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eede480-ca51-42f6-8c7b-a21769018b92-istio-podinfo\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.992894 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.992870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-istio-token\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:52.993103 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:52.993083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5kgg\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-kube-api-access-v5kgg\") pod \"e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") " pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:53.086265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:53.086163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:53.160878 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:53.160850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:53.368579 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:53.368289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"]
Sep 10 23:25:53.373583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:53.373517    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:25:53.373680 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:53.373632    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:25:53.373719 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:53.373674    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:25:53.777844 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:53.777757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" event={"ID":"7eede480-ca51-42f6-8c7b-a21769018b92","Type":"ContainerStarted","Data":"4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68"}
Sep 10 23:25:53.777844 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:53.777810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" event={"ID":"7eede480-ca51-42f6-8c7b-a21769018b92","Type":"ContainerStarted","Data":"8cb2cafbae86143e4dc0001bd50d45adf6c885917ec89a5353a34ee4f507dd13"}
Sep 10 23:25:53.797718 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:53.797670    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" podStartSLOduration=1.797652429 podStartE2EDuration="1.797652429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:25:53.795020351 +0000 UTC m=+1453.496653293" watchObservedRunningTime="2026-09-10 23:25:53.797652429 +0000 UTC m=+1453.499285372"
Sep 10 23:25:54.086917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:54.086835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:54.092789 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:54.092766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:54.783910 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:54.783876    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:54.784995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:54.784976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:25:55.687591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:55.687561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:55.780327 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:55.780288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:56.828366 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:56.828331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:56.928533 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:56.928499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:57.029037 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:57.029006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:57.331081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:57.331048    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:57.385510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:25:57.385473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:04.565741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:04.565707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:08.974182 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:08.974135    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:26:13.943812 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:13.943776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:17.705506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:17.705462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:18.949986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:18.949955    2580 generic.go:356] "Generic (PLEG): container finished" podID="7eede480-ca51-42f6-8c7b-a21769018b92" containerID="4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68" exitCode=0
Sep 10 23:26:18.950381 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:18.950021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" event={"ID":"7eede480-ca51-42f6-8c7b-a21769018b92","Type":"ContainerDied","Data":"4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68"}
Sep 10 23:26:18.950457 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:18.950443    2580 scope.go:117] "RemoveContainer" containerID="4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68"
Sep 10 23:26:18.951733 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:18.951703    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:26:18.951833 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:18.951769    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:26:18.951833 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:18.951798    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:26:19.958486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:19.958382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" event={"ID":"7eede480-ca51-42f6-8c7b-a21769018b92","Type":"ContainerStarted","Data":"c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031"}
Sep 10 23:26:20.086689 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:20.086657    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:26:20.087945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:20.087913    2580 patch_prober.go:28] interesting pod/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 10 23:26:20.088086 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:20.087964    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 10 23:26:20.617243 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:20.617210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:21.045895 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.045774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:21.086484 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.086456    2580 patch_prober.go:28] interesting pod/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 10 23:26:21.086666 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.086511    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 10 23:26:21.312460 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.312382    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"]
Sep 10 23:26:21.329071 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.329021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.338605 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.333653    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-72f076-openshift-default-dockercfg-4mnll\""
Sep 10 23:26:21.338605 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.337168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"]
Sep 10 23:26:21.371169 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.371102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:21.380193 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.380164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-data\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.380338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.380219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-credential-socket\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.380338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.380275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-certs\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.380338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.380312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75184e59-63f8-418f-907a-265e873534dd-istio-podinfo\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.380439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.380356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75184e59-63f8-418f-907a-265e873534dd-istiod-ca-cert\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.380439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.380381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-socket\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.380505 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.380447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-envoy\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.380505 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.380486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-istio-token\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.380605 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.380590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnj29\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-kube-api-access-tnj29\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.482190 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.482148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75184e59-63f8-418f-907a-265e873534dd-istiod-ca-cert\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.482470 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.482451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-socket\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.482651 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.482633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-envoy\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.482782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.482768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-istio-token\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.482912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.482898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnj29\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-kube-api-access-tnj29\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.483081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.483064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-data\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.483215 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.483198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-credential-socket\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.483341 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.483328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-certs\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.483459 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.483446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75184e59-63f8-418f-907a-265e873534dd-istio-podinfo\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.485069 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.485045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75184e59-63f8-418f-907a-265e873534dd-istiod-ca-cert\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.486227 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.485464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-data\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.486376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.485858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-socket\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.486496 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.485952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-credential-socket\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.486614 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.486169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-certs\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.494910 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.494874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-envoy\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.495608 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.495580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75184e59-63f8-418f-907a-265e873534dd-istio-podinfo\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.496045 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.496024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-istio-token\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.497306 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.497286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnj29\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-kube-api-access-tnj29\") pod \"e2e-api-b-72f076-openshift-default-5699c8958b-whvfq\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") " pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.676258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.676221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:21.682628 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.682601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:21.984502 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.984476    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"]
Sep 10 23:26:21.985905 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:26:21.985879    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75184e59_63f8_418f_907a_265e873534dd.slice/crio-97a2721da7aece4be386b352f824c2e847277282d29d5dc2c8fd7830c5c8c9f7 WatchSource:0}: Error finding container 97a2721da7aece4be386b352f824c2e847277282d29d5dc2c8fd7830c5c8c9f7: Status 404 returned error can't find the container with id 97a2721da7aece4be386b352f824c2e847277282d29d5dc2c8fd7830c5c8c9f7
Sep 10 23:26:21.988308 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.988273    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:26:21.988394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.988349    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:26:21.988394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:21.988380    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:26:22.086740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:22.086708    2580 patch_prober.go:28] interesting pod/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 10 23:26:22.087080 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:22.086768    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 10 23:26:22.977911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:22.977878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" event={"ID":"75184e59-63f8-418f-907a-265e873534dd","Type":"ContainerStarted","Data":"6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6"}
Sep 10 23:26:22.978095 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:22.977930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" event={"ID":"75184e59-63f8-418f-907a-265e873534dd","Type":"ContainerStarted","Data":"97a2721da7aece4be386b352f824c2e847277282d29d5dc2c8fd7830c5c8c9f7"}
Sep 10 23:26:23.000313 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:23.000265    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" podStartSLOduration=2.000250411 podStartE2EDuration="2.000250411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:26:22.997560203 +0000 UTC m=+1482.699193137" watchObservedRunningTime="2026-09-10 23:26:23.000250411 +0000 UTC m=+1482.701883365"
Sep 10 23:26:23.087219 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:23.087190    2580 patch_prober.go:28] interesting pod/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 10 23:26:23.087658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:23.087633    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 10 23:26:23.676813 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:23.676775    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:23.681839 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:23.681814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:23.985031 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:23.984946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:23.985919 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:23.985897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:24.087360 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:24.087328    2580 patch_prober.go:28] interesting pod/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 10 23:26:24.087856 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:24.087383    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 10 23:26:24.784162 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:24.784127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:26:25.087113 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:25.087026    2580 patch_prober.go:28] interesting pod/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 10 23:26:25.087113 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:25.087079    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 10 23:26:25.746266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:25.746199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:25.849289 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:25.849250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:26.040375 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:26.040251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:26.087396 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:26.087364    2580 patch_prober.go:28] interesting pod/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 10 23:26:26.087593 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:26.087414    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 10 23:26:26.135043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:26.135011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:26.510716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:26.510671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:26.612416 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:26.612388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:26.818472 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:26.818361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:27.102750 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:27.102719    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:26:27.104805 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:27.104775    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:26:39.184510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:39.184469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:26:43.256409 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:43.256368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:45.051566 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:45.039474    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"]
Sep 10 23:26:45.053867 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:45.053736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:49.785262 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:49.785189    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:50.157347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:50.157316    2580 generic.go:356] "Generic (PLEG): container finished" podID="75184e59-63f8-418f-907a-265e873534dd" containerID="6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6" exitCode=0
Sep 10 23:26:50.157516 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:50.157386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" event={"ID":"75184e59-63f8-418f-907a-265e873534dd","Type":"ContainerDied","Data":"6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6"}
Sep 10 23:26:50.157798 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:50.157785    2580 scope.go:117] "RemoveContainer" containerID="6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6"
Sep 10 23:26:50.158998 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:50.158972    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:26:50.159081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:50.159034    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:26:50.159081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:50.159066    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:26:51.165582 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:51.165530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" event={"ID":"75184e59-63f8-418f-907a-265e873534dd","Type":"ContainerStarted","Data":"0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982"}
Sep 10 23:26:51.676851 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:51.676817    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:51.678177 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:51.678147    2580 patch_prober.go:28] interesting pod/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 23:26:51.678306 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:51.678199    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 23:26:52.672730 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:52.672702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:52.676807 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:52.676783    2580 patch_prober.go:28] interesting pod/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 23:26:52.676901 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:52.676836    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 23:26:53.677069 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:53.677031    2580 patch_prober.go:28] interesting pod/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 23:26:53.677450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:53.677092    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 23:26:53.985433 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:53.985339    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:54.676594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:54.676536    2580 patch_prober.go:28] interesting pod/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 23:26:54.676773 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:54.676615    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 23:26:55.676864 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:55.676827    2580 patch_prober.go:28] interesting pod/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 23:26:55.677248 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:55.676886    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 23:26:56.680581 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:56.680530    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:26:56.681783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:26:56.681763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:27:00.080342 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.080299    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525" podUID="f6fa7cc3-d57c-409e-94a5-89379502205b" containerName="main" containerID="cri-o://7d8bb3a1d1af1de868b61b6d8701c8e7797e4362f0982de77647239e737228a3" gracePeriod=45
Sep 10 23:27:00.227489 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.227458    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6fa7cc3-d57c-409e-94a5-89379502205b" containerID="7d8bb3a1d1af1de868b61b6d8701c8e7797e4362f0982de77647239e737228a3" exitCode=0
Sep 10 23:27:00.227760 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.227584    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525" event={"ID":"f6fa7cc3-d57c-409e-94a5-89379502205b","Type":"ContainerDied","Data":"7d8bb3a1d1af1de868b61b6d8701c8e7797e4362f0982de77647239e737228a3"}
Sep 10 23:27:00.386527 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.386484    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:27:00.532726 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.532691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fa7cc3-d57c-409e-94a5-89379502205b-tls-certs\") pod \"f6fa7cc3-d57c-409e-94a5-89379502205b\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") "
Sep 10 23:27:00.532726 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.532731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-dshm\") pod \"f6fa7cc3-d57c-409e-94a5-89379502205b\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") "
Sep 10 23:27:00.533014 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.532778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-tmp-dir\") pod \"f6fa7cc3-d57c-409e-94a5-89379502205b\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") "
Sep 10 23:27:00.533014 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.532833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-home\") pod \"f6fa7cc3-d57c-409e-94a5-89379502205b\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") "
Sep 10 23:27:00.533014 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.532892    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-model-cache\") pod \"f6fa7cc3-d57c-409e-94a5-89379502205b\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") "
Sep 10 23:27:00.533014 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.532940    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j658r\" (UniqueName: \"kubernetes.io/projected/f6fa7cc3-d57c-409e-94a5-89379502205b-kube-api-access-j658r\") pod \"f6fa7cc3-d57c-409e-94a5-89379502205b\" (UID: \"f6fa7cc3-d57c-409e-94a5-89379502205b\") "
Sep 10 23:27:00.533257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.533061    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6fa7cc3-d57c-409e-94a5-89379502205b" (UID: "f6fa7cc3-d57c-409e-94a5-89379502205b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:00.533257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.533122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-home" (OuterVolumeSpecName: "home") pod "f6fa7cc3-d57c-409e-94a5-89379502205b" (UID: "f6fa7cc3-d57c-409e-94a5-89379502205b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:00.533257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.533131    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6fa7cc3-d57c-409e-94a5-89379502205b" (UID: "f6fa7cc3-d57c-409e-94a5-89379502205b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:00.533438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.533411    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-model-cache\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:00.533499 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.533451    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-tmp-dir\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:00.533499 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.533464    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-home\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:00.535341 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.535303    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-dshm" (OuterVolumeSpecName: "dshm") pod "f6fa7cc3-d57c-409e-94a5-89379502205b" (UID: "f6fa7cc3-d57c-409e-94a5-89379502205b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:00.535485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.535366    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fa7cc3-d57c-409e-94a5-89379502205b-kube-api-access-j658r" (OuterVolumeSpecName: "kube-api-access-j658r") pod "f6fa7cc3-d57c-409e-94a5-89379502205b" (UID: "f6fa7cc3-d57c-409e-94a5-89379502205b"). InnerVolumeSpecName "kube-api-access-j658r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:00.535485 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.535372    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6fa7cc3-d57c-409e-94a5-89379502205b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6fa7cc3-d57c-409e-94a5-89379502205b" (UID: "f6fa7cc3-d57c-409e-94a5-89379502205b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:27:00.635042 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.634947    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j658r\" (UniqueName: \"kubernetes.io/projected/f6fa7cc3-d57c-409e-94a5-89379502205b-kube-api-access-j658r\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:00.635042 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.634974    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fa7cc3-d57c-409e-94a5-89379502205b-tls-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:00.635042 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:00.634985    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fa7cc3-d57c-409e-94a5-89379502205b-dshm\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:01.235226 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:01.235199    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"
Sep 10 23:27:01.235731 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:01.235243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525" event={"ID":"f6fa7cc3-d57c-409e-94a5-89379502205b","Type":"ContainerDied","Data":"1fb678cca5ef2ed0f59443a06bb2fb543245712d7c94947c01a22b1b09f9f431"}
Sep 10 23:27:01.235731 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:01.235311    2580 scope.go:117] "RemoveContainer" containerID="7d8bb3a1d1af1de868b61b6d8701c8e7797e4362f0982de77647239e737228a3"
Sep 10 23:27:01.250414 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:01.250383    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"]
Sep 10 23:27:01.254282 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:01.254258    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-a-d74b488b-kserve-74b6857b4b-47525"]
Sep 10 23:27:02.834907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:02.834872    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6fa7cc3-d57c-409e-94a5-89379502205b" path="/var/lib/kubelet/pods/f6fa7cc3-d57c-409e-94a5-89379502205b/volumes"
Sep 10 23:27:08.087649 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:08.087609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:08.143467 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:08.143406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:08.244263 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:08.244231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:08.343211 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:08.343131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:08.604808 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:08.604775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:14.559349 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:14.559312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:14.615400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:14.615367    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:14.796379 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:14.796347    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"]
Sep 10 23:27:14.796625 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:14.796597    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy" containerID="cri-o://c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031" gracePeriod=30
Sep 10 23:27:14.815968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:14.815888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:15.265741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:15.265705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:16.673861 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:16.673827    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:27:20.056357 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.056329    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:27:20.137458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137421    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-credential-socket\") pod \"7eede480-ca51-42f6-8c7b-a21769018b92\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") "
Sep 10 23:27:20.137674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137487    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-socket\") pod \"7eede480-ca51-42f6-8c7b-a21769018b92\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") "
Sep 10 23:27:20.137674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eede480-ca51-42f6-8c7b-a21769018b92-istiod-ca-cert\") pod \"7eede480-ca51-42f6-8c7b-a21769018b92\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") "
Sep 10 23:27:20.137674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-data\") pod \"7eede480-ca51-42f6-8c7b-a21769018b92\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") "
Sep 10 23:27:20.137674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137590    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-certs\") pod \"7eede480-ca51-42f6-8c7b-a21769018b92\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") "
Sep 10 23:27:20.137674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137619    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5kgg\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-kube-api-access-v5kgg\") pod \"7eede480-ca51-42f6-8c7b-a21769018b92\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") "
Sep 10 23:27:20.137674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-istio-token\") pod \"7eede480-ca51-42f6-8c7b-a21769018b92\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") "
Sep 10 23:27:20.138010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137734    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-envoy\") pod \"7eede480-ca51-42f6-8c7b-a21769018b92\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") "
Sep 10 23:27:20.138010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137756    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7eede480-ca51-42f6-8c7b-a21769018b92" (UID: "7eede480-ca51-42f6-8c7b-a21769018b92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:20.138010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137790    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eede480-ca51-42f6-8c7b-a21769018b92-istio-podinfo\") pod \"7eede480-ca51-42f6-8c7b-a21769018b92\" (UID: \"7eede480-ca51-42f6-8c7b-a21769018b92\") "
Sep 10 23:27:20.138010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-data" (OuterVolumeSpecName: "istio-data") pod "7eede480-ca51-42f6-8c7b-a21769018b92" (UID: "7eede480-ca51-42f6-8c7b-a21769018b92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:20.138010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137884    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7eede480-ca51-42f6-8c7b-a21769018b92" (UID: "7eede480-ca51-42f6-8c7b-a21769018b92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:20.138010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.137954    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eede480-ca51-42f6-8c7b-a21769018b92-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7eede480-ca51-42f6-8c7b-a21769018b92" (UID: "7eede480-ca51-42f6-8c7b-a21769018b92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:20.138305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.138049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7eede480-ca51-42f6-8c7b-a21769018b92" (UID: "7eede480-ca51-42f6-8c7b-a21769018b92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:20.138374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.138328    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:20.138374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.138349    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:20.138374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.138363    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eede480-ca51-42f6-8c7b-a21769018b92-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:20.138562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.138375    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:20.138562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.138387    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:20.140220 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.140185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-kube-api-access-v5kgg" (OuterVolumeSpecName: "kube-api-access-v5kgg") pod "7eede480-ca51-42f6-8c7b-a21769018b92" (UID: "7eede480-ca51-42f6-8c7b-a21769018b92"). InnerVolumeSpecName "kube-api-access-v5kgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:20.140346 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.140267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7eede480-ca51-42f6-8c7b-a21769018b92" (UID: "7eede480-ca51-42f6-8c7b-a21769018b92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:20.140623 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.140593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eede480-ca51-42f6-8c7b-a21769018b92-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7eede480-ca51-42f6-8c7b-a21769018b92" (UID: "7eede480-ca51-42f6-8c7b-a21769018b92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:27:20.140729 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.140623    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-istio-token" (OuterVolumeSpecName: "istio-token") pod "7eede480-ca51-42f6-8c7b-a21769018b92" (UID: "7eede480-ca51-42f6-8c7b-a21769018b92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:20.240019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.239932    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eede480-ca51-42f6-8c7b-a21769018b92-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:20.240019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.239960    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5kgg\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-kube-api-access-v5kgg\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:20.240019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.239972    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eede480-ca51-42f6-8c7b-a21769018b92-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:20.240019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.239982    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eede480-ca51-42f6-8c7b-a21769018b92-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:20.360961 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.360928    2580 generic.go:356] "Generic (PLEG): container finished" podID="7eede480-ca51-42f6-8c7b-a21769018b92" containerID="c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031" exitCode=0
Sep 10 23:27:20.361151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.360974    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"
Sep 10 23:27:20.361151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.361008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" event={"ID":"7eede480-ca51-42f6-8c7b-a21769018b92","Type":"ContainerDied","Data":"c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031"}
Sep 10 23:27:20.361151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.361072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb" event={"ID":"7eede480-ca51-42f6-8c7b-a21769018b92","Type":"ContainerDied","Data":"8cb2cafbae86143e4dc0001bd50d45adf6c885917ec89a5353a34ee4f507dd13"}
Sep 10 23:27:20.361151 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.361103    2580 scope.go:117] "RemoveContainer" containerID="c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031"
Sep 10 23:27:20.371091 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.371077    2580 scope.go:117] "RemoveContainer" containerID="4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68"
Sep 10 23:27:20.379595 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.379578    2580 scope.go:117] "RemoveContainer" containerID="c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031"
Sep 10 23:27:20.379863 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:27:20.379844    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031\": container with ID starting with c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031 not found: ID does not exist" containerID="c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031"
Sep 10 23:27:20.379913 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.379872    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031"} err="failed to get container status \"c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031\": rpc error: code = NotFound desc = could not find container \"c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031\": container with ID starting with c4405456abbf93958dc3a7290a9f3333b05b11628335c7976b12f945cbcfe031 not found: ID does not exist"
Sep 10 23:27:20.379913 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.379890    2580 scope.go:117] "RemoveContainer" containerID="4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68"
Sep 10 23:27:20.380151 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:27:20.380133    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68\": container with ID starting with 4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68 not found: ID does not exist" containerID="4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68"
Sep 10 23:27:20.380202 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.380157    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68"} err="failed to get container status \"4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68\": rpc error: code = NotFound desc = could not find container \"4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68\": container with ID starting with 4979956a3d7963facc7e82235cadd1abb9dcf866368c00862b34a78e9a90ba68 not found: ID does not exist"
Sep 10 23:27:20.384160 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.384140    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"]
Sep 10 23:27:20.387874 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.387854    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a4c77c-openshift-default-944d679c4-s8cxb"]
Sep 10 23:27:20.834512 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:20.834476    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" path="/var/lib/kubelet/pods/7eede480-ca51-42f6-8c7b-a21769018b92/volumes"
Sep 10 23:27:30.680703 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:30.680668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:31.885225 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:31.885195    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:37.001789 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:37.001756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:37.057328 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:37.057298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:37.235751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:37.235722    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"]
Sep 10 23:27:37.236001 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:37.235976    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy" containerID="cri-o://0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982" gracePeriod=30
Sep 10 23:27:37.248409 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:37.248378    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:37.671372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:37.671327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:41.683347 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:41.683306    2580 patch_prober.go:28] interesting pod/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 23:27:41.683752 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:41.683371    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:27:42.490820 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.490797    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:27:42.502463 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.502388    2580 generic.go:356] "Generic (PLEG): container finished" podID="75184e59-63f8-418f-907a-265e873534dd" containerID="0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982" exitCode=0
Sep 10 23:27:42.502463 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.502436    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"
Sep 10 23:27:42.502662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.502474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" event={"ID":"75184e59-63f8-418f-907a-265e873534dd","Type":"ContainerDied","Data":"0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982"}
Sep 10 23:27:42.502662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.502555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq" event={"ID":"75184e59-63f8-418f-907a-265e873534dd","Type":"ContainerDied","Data":"97a2721da7aece4be386b352f824c2e847277282d29d5dc2c8fd7830c5c8c9f7"}
Sep 10 23:27:42.502662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.502560    2580 scope.go:117] "RemoveContainer" containerID="0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982"
Sep 10 23:27:42.518177 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.518155    2580 scope.go:117] "RemoveContainer" containerID="6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6"
Sep 10 23:27:42.529195 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.529173    2580 scope.go:117] "RemoveContainer" containerID="0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982"
Sep 10 23:27:42.529488 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:27:42.529470    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982\": container with ID starting with 0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982 not found: ID does not exist" containerID="0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982"
Sep 10 23:27:42.529585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.529497    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982"} err="failed to get container status \"0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982\": rpc error: code = NotFound desc = could not find container \"0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982\": container with ID starting with 0b8a55b5d2d7dec635e7198d86e668a7f6a358f9653dcbc2592bf89a4cc28982 not found: ID does not exist"
Sep 10 23:27:42.529585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.529516    2580 scope.go:117] "RemoveContainer" containerID="6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6"
Sep 10 23:27:42.529833 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:27:42.529814    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6\": container with ID starting with 6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6 not found: ID does not exist" containerID="6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6"
Sep 10 23:27:42.529903 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.529840    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6"} err="failed to get container status \"6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6\": rpc error: code = NotFound desc = could not find container \"6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6\": container with ID starting with 6fcca77ed3408f016fee95aa3fcf8b5b65a15e5b586639a8e5d3a56c43248ce6 not found: ID does not exist"
Sep 10 23:27:42.672823 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.672792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnj29\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-kube-api-access-tnj29\") pod \"75184e59-63f8-418f-907a-265e873534dd\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") "
Sep 10 23:27:42.672995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.672840    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75184e59-63f8-418f-907a-265e873534dd-istio-podinfo\") pod \"75184e59-63f8-418f-907a-265e873534dd\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") "
Sep 10 23:27:42.672995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.672913    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-credential-socket\") pod \"75184e59-63f8-418f-907a-265e873534dd\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") "
Sep 10 23:27:42.672995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.672953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-certs\") pod \"75184e59-63f8-418f-907a-265e873534dd\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") "
Sep 10 23:27:42.672995 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.672984    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-envoy\") pod \"75184e59-63f8-418f-907a-265e873534dd\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") "
Sep 10 23:27:42.673206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673045    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-socket\") pod \"75184e59-63f8-418f-907a-265e873534dd\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") "
Sep 10 23:27:42.673206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-data\") pod \"75184e59-63f8-418f-907a-265e873534dd\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") "
Sep 10 23:27:42.673206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673139    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-istio-token\") pod \"75184e59-63f8-418f-907a-265e873534dd\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") "
Sep 10 23:27:42.673206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75184e59-63f8-418f-907a-265e873534dd-istiod-ca-cert\") pod \"75184e59-63f8-418f-907a-265e873534dd\" (UID: \"75184e59-63f8-418f-907a-265e873534dd\") "
Sep 10 23:27:42.673400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75184e59-63f8-418f-907a-265e873534dd" (UID: "75184e59-63f8-418f-907a-265e873534dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:42.673400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673301    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75184e59-63f8-418f-907a-265e873534dd" (UID: "75184e59-63f8-418f-907a-265e873534dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:42.673400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75184e59-63f8-418f-907a-265e873534dd" (UID: "75184e59-63f8-418f-907a-265e873534dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:42.673569 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673482    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "75184e59-63f8-418f-907a-265e873534dd" (UID: "75184e59-63f8-418f-907a-265e873534dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:42.673632 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75184e59-63f8-418f-907a-265e873534dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75184e59-63f8-418f-907a-265e873534dd" (UID: "75184e59-63f8-418f-907a-265e873534dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:42.673687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673647    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:42.673687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673669    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:42.673687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673684    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:42.673810 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.673698    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:42.675316 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.675288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75184e59-63f8-418f-907a-265e873534dd" (UID: "75184e59-63f8-418f-907a-265e873534dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:27:42.675488 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.675470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "75184e59-63f8-418f-907a-265e873534dd" (UID: "75184e59-63f8-418f-907a-265e873534dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:42.675600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.675584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75184e59-63f8-418f-907a-265e873534dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75184e59-63f8-418f-907a-265e873534dd" (UID: "75184e59-63f8-418f-907a-265e873534dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:27:42.675668 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.675643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-kube-api-access-tnj29" (OuterVolumeSpecName: "kube-api-access-tnj29") pod "75184e59-63f8-418f-907a-265e873534dd" (UID: "75184e59-63f8-418f-907a-265e873534dd"). InnerVolumeSpecName "kube-api-access-tnj29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:42.774394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.774317    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:42.774394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.774344    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75184e59-63f8-418f-907a-265e873534dd-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:42.774394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.774353    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnj29\" (UniqueName: \"kubernetes.io/projected/75184e59-63f8-418f-907a-265e873534dd-kube-api-access-tnj29\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:42.774394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.774361    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75184e59-63f8-418f-907a-265e873534dd-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:42.774394 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.774371    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75184e59-63f8-418f-907a-265e873534dd-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:42.825209 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.825155    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"]
Sep 10 23:27:42.827486 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.827464    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-72f076-openshift-default-5699c8958b-whvfq"]
Sep 10 23:27:42.834097 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:42.834074    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75184e59-63f8-418f-907a-265e873534dd" path="/var/lib/kubelet/pods/75184e59-63f8-418f-907a-265e873534dd/volumes"
Sep 10 23:27:46.676570 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:46.676522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:27:47.573555 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:47.573517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:47.667666 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:47.667637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:48.176572 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.176517    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"]
Sep 10 23:27:48.177410 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.177297    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj" podUID="adcea5dc-ad08-451d-b5ae-01ea0b6da77b" containerName="maas-api" containerID="cri-o://7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb" gracePeriod=30
Sep 10 23:27:48.188629 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.188597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:48.258066 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.258018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:48.313878 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.313845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:48.388793 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.388766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:48.442474 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.442383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:48.448945 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.448915    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:27:48.532584 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.531711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-maas-api-tls\") pod \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") "
Sep 10 23:27:48.532584 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.531786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-openshift-service-ca\") pod \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") "
Sep 10 23:27:48.532584 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.531823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbsx8\" (UniqueName: \"kubernetes.io/projected/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-kube-api-access-kbsx8\") pod \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") "
Sep 10 23:27:48.532584 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.531962    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-metrics-certs\") pod \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\" (UID: \"adcea5dc-ad08-451d-b5ae-01ea0b6da77b\") "
Sep 10 23:27:48.533470 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.533229    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adcea5dc-ad08-451d-b5ae-01ea0b6da77b" (UID: "adcea5dc-ad08-451d-b5ae-01ea0b6da77b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:48.537127 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.536956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "adcea5dc-ad08-451d-b5ae-01ea0b6da77b" (UID: "adcea5dc-ad08-451d-b5ae-01ea0b6da77b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:27:48.537127 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.537114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adcea5dc-ad08-451d-b5ae-01ea0b6da77b" (UID: "adcea5dc-ad08-451d-b5ae-01ea0b6da77b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:27:48.539954 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.539927    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-kube-api-access-kbsx8" (OuterVolumeSpecName: "kube-api-access-kbsx8") pod "adcea5dc-ad08-451d-b5ae-01ea0b6da77b" (UID: "adcea5dc-ad08-451d-b5ae-01ea0b6da77b"). InnerVolumeSpecName "kube-api-access-kbsx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:48.543959 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.542623    2580 generic.go:356] "Generic (PLEG): container finished" podID="adcea5dc-ad08-451d-b5ae-01ea0b6da77b" containerID="7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb" exitCode=0
Sep 10 23:27:48.543959 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.542671    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"
Sep 10 23:27:48.543959 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.542723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj" event={"ID":"adcea5dc-ad08-451d-b5ae-01ea0b6da77b","Type":"ContainerDied","Data":"7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb"}
Sep 10 23:27:48.543959 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.542764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj" event={"ID":"adcea5dc-ad08-451d-b5ae-01ea0b6da77b","Type":"ContainerDied","Data":"c493228485271475d012d99bd9b0ffafdac5e4a8ea11b16e26f4438d784974b3"}
Sep 10 23:27:48.543959 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.542792    2580 scope.go:117] "RemoveContainer" containerID="7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb"
Sep 10 23:27:48.566834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.566804    2580 scope.go:117] "RemoveContainer" containerID="7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb"
Sep 10 23:27:48.567231 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:27:48.567196    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb\": container with ID starting with 7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb not found: ID does not exist" containerID="7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb"
Sep 10 23:27:48.567336 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.567243    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb"} err="failed to get container status \"7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb\": rpc error: code = NotFound desc = could not find container \"7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb\": container with ID starting with 7a56cf7e2308cf9cdec3b1f577d739b6ae222bd41c3ce30e6c7a80e97211a6eb not found: ID does not exist"
Sep 10 23:27:48.580435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.580406    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"]
Sep 10 23:27:48.582883 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.582851    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bda598-669d5b56db-7wmzj"]
Sep 10 23:27:48.633169 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.633124    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:48.633169 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.633165    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:48.633169 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.633180    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:48.633493 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.633195    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsx8\" (UniqueName: \"kubernetes.io/projected/adcea5dc-ad08-451d-b5ae-01ea0b6da77b-kube-api-access-kbsx8\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:27:48.722494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.722392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:48.834624 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:48.834593    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adcea5dc-ad08-451d-b5ae-01ea0b6da77b" path="/var/lib/kubelet/pods/adcea5dc-ad08-451d-b5ae-01ea0b6da77b/volumes"
Sep 10 23:27:54.799423 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:54.799390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:54.900098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:54.900062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:55.028617 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:55.028584    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"]
Sep 10 23:27:55.028894 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:55.028867    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy" containerID="cri-o://425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5" gracePeriod=30
Sep 10 23:27:55.052475 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:55.052398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:55.558595 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:55.558560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:57.015179 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:27:57.015146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:00.296971 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.296905    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:28:00.349932 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.349897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n44sr\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-kube-api-access-n44sr\") pod \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") "
Sep 10 23:28:00.350071 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.349956    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-token\") pod \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") "
Sep 10 23:28:00.350071 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istiod-ca-cert\") pod \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") "
Sep 10 23:28:00.350152 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-data\") pod \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") "
Sep 10 23:28:00.350152 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-credential-socket\") pod \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") "
Sep 10 23:28:00.350233 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350175    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-podinfo\") pod \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") "
Sep 10 23:28:00.350233 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350208    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-certs\") pod \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") "
Sep 10 23:28:00.350336 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-socket\") pod \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") "
Sep 10 23:28:00.350336 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350290    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-envoy\") pod \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\" (UID: \"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d\") "
Sep 10 23:28:00.350445 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" (UID: "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:28:00.350445 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" (UID: "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:00.350611 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-data" (OuterVolumeSpecName: "istio-data") pod "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" (UID: "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:00.350728 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" (UID: "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:00.350842 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350821    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" (UID: "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:00.350897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350827    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:00.350897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350864    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:00.350897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350878    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:00.350897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.350892    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:00.352583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.352522    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-kube-api-access-n44sr" (OuterVolumeSpecName: "kube-api-access-n44sr") pod "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" (UID: "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d"). InnerVolumeSpecName "kube-api-access-n44sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:00.352935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.352913    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-token" (OuterVolumeSpecName: "istio-token") pod "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" (UID: "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:00.353279 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.353258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" (UID: "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:28:00.353339 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.353314    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" (UID: "25a7d0cd-a7db-4fe3-991c-3e4e5959b43d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:00.451716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.451675    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:00.451716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.451704    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:00.451716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.451716    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:00.451952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.451732    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n44sr\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-kube-api-access-n44sr\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:00.451952 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.451742    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:00.612989 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.612954    2580 generic.go:356] "Generic (PLEG): container finished" podID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerID="425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5" exitCode=0
Sep 10 23:28:00.613155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.613003    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"
Sep 10 23:28:00.613155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.613030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" event={"ID":"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d","Type":"ContainerDied","Data":"425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5"}
Sep 10 23:28:00.613155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.613081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg" event={"ID":"25a7d0cd-a7db-4fe3-991c-3e4e5959b43d","Type":"ContainerDied","Data":"a7ce587448fbeadb2f6c84ee1095b6a4e1c14f18e2082eb210f4392f94692ed4"}
Sep 10 23:28:00.613155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.613107    2580 scope.go:117] "RemoveContainer" containerID="425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5"
Sep 10 23:28:00.623695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.623651    2580 scope.go:117] "RemoveContainer" containerID="5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a"
Sep 10 23:28:00.633013 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.632995    2580 scope.go:117] "RemoveContainer" containerID="425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5"
Sep 10 23:28:00.633299 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:28:00.633275    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5\": container with ID starting with 425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5 not found: ID does not exist" containerID="425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5"
Sep 10 23:28:00.633400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.633306    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5"} err="failed to get container status \"425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5\": rpc error: code = NotFound desc = could not find container \"425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5\": container with ID starting with 425ae4e4d156f3fa92c4b2ebbac5281689826ba702ec7510ddcbc969234331d5 not found: ID does not exist"
Sep 10 23:28:00.633400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.633323    2580 scope.go:117] "RemoveContainer" containerID="5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a"
Sep 10 23:28:00.633582 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:28:00.633566    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a\": container with ID starting with 5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a not found: ID does not exist" containerID="5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a"
Sep 10 23:28:00.633636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.633588    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a"} err="failed to get container status \"5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a\": rpc error: code = NotFound desc = could not find container \"5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a\": container with ID starting with 5ddb69f2829ed8461133e6b67d28031893e59a481b6ab0504cc421418b391c4a not found: ID does not exist"
Sep 10 23:28:00.634636 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.634613    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"]
Sep 10 23:28:00.638232 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.638211    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bda598-openshift-default-77b79bd476-hdhhg"]
Sep 10 23:28:00.833693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:00.833652    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" path="/var/lib/kubelet/pods/25a7d0cd-a7db-4fe3-991c-3e4e5959b43d/volumes"
Sep 10 23:28:05.511595 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:05.511563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:06.151610 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:06.151577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:06.252241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:06.252208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:06.923255 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:06.923222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:07.237644 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:07.237559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:13.181519 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:13.181487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:13.238463 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:13.238425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:13.412465 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:13.412432    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"]
Sep 10 23:28:13.412753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:13.412649    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy" containerID="cri-o://a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b" gracePeriod=30
Sep 10 23:28:13.428123 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:13.428093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:13.870582 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:13.870535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:28:16.004859 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.004827    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7846-cc9769"]
Sep 10 23:28:16.005237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005224    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy"
Sep 10 23:28:16.005282 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005238    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy"
Sep 10 23:28:16.005282 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005251    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy"
Sep 10 23:28:16.005282 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005256    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy"
Sep 10 23:28:16.005282 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005267    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adcea5dc-ad08-451d-b5ae-01ea0b6da77b" containerName="maas-api"
Sep 10 23:28:16.005282 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005272    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adcea5dc-ad08-451d-b5ae-01ea0b6da77b" containerName="maas-api"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005284    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6fa7cc3-d57c-409e-94a5-89379502205b" containerName="main"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005289    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6fa7cc3-d57c-409e-94a5-89379502205b" containerName="main"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005297    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005302    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005310    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005315    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005371    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="adcea5dc-ad08-451d-b5ae-01ea0b6da77b" containerName="maas-api"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005379    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005385    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6fa7cc3-d57c-409e-94a5-89379502205b" containerName="main"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005391    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy"
Sep 10 23:28:16.005435 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.005396    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy"
Sep 10 23:28:16.008356 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.008342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-cc9769"
Sep 10 23:28:16.010770 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.010747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4p5h\""
Sep 10 23:28:16.013177 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.013154    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7846-cc9769"]
Sep 10 23:28:16.113154 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.113123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rfjw\" (UniqueName: \"kubernetes.io/projected/a9fb65b0-20b8-44ca-94a8-67a756dfa3a9-kube-api-access-4rfjw\") pod \"test-curl-7846-cc9769\" (UID: \"a9fb65b0-20b8-44ca-94a8-67a756dfa3a9\") " pod="opendatahub/test-curl-7846-cc9769"
Sep 10 23:28:16.214613 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.214576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rfjw\" (UniqueName: \"kubernetes.io/projected/a9fb65b0-20b8-44ca-94a8-67a756dfa3a9-kube-api-access-4rfjw\") pod \"test-curl-7846-cc9769\" (UID: \"a9fb65b0-20b8-44ca-94a8-67a756dfa3a9\") " pod="opendatahub/test-curl-7846-cc9769"
Sep 10 23:28:16.222489 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.222461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rfjw\" (UniqueName: \"kubernetes.io/projected/a9fb65b0-20b8-44ca-94a8-67a756dfa3a9-kube-api-access-4rfjw\") pod \"test-curl-7846-cc9769\" (UID: \"a9fb65b0-20b8-44ca-94a8-67a756dfa3a9\") " pod="opendatahub/test-curl-7846-cc9769"
Sep 10 23:28:16.319596 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.319484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-cc9769"
Sep 10 23:28:16.594162 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.594131    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7846-cc9769"]
Sep 10 23:28:16.595614 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:28:16.595586    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9fb65b0_20b8_44ca_94a8_67a756dfa3a9.slice/crio-2bd7a7b18d571e92de9c468d4ba6b47f7b5b143ab957a518e360670dcde80bd3 WatchSource:0}: Error finding container 2bd7a7b18d571e92de9c468d4ba6b47f7b5b143ab957a518e360670dcde80bd3: Status 404 returned error can't find the container with id 2bd7a7b18d571e92de9c468d4ba6b47f7b5b143ab957a518e360670dcde80bd3
Sep 10 23:28:16.597457 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.597441    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:28:16.711058 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:16.711019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7846-cc9769" event={"ID":"a9fb65b0-20b8-44ca-94a8-67a756dfa3a9","Type":"ContainerStarted","Data":"2bd7a7b18d571e92de9c468d4ba6b47f7b5b143ab957a518e360670dcde80bd3"}
Sep 10 23:28:18.914013 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.913989    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:28:18.940631 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.940561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-certs\") pod \"3d952917-3a68-4659-8674-52eb7a85f4b3\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") "
Sep 10 23:28:18.940631 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.940623    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-podinfo\") pod \"3d952917-3a68-4659-8674-52eb7a85f4b3\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") "
Sep 10 23:28:18.940891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.940655    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-data\") pod \"3d952917-3a68-4659-8674-52eb7a85f4b3\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") "
Sep 10 23:28:18.940891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.940710    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-socket\") pod \"3d952917-3a68-4659-8674-52eb7a85f4b3\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") "
Sep 10 23:28:18.940891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.940783    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8k6c\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-kube-api-access-t8k6c\") pod \"3d952917-3a68-4659-8674-52eb7a85f4b3\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") "
Sep 10 23:28:18.940891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.940823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-envoy\") pod \"3d952917-3a68-4659-8674-52eb7a85f4b3\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") "
Sep 10 23:28:18.940891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.940870    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-credential-socket\") pod \"3d952917-3a68-4659-8674-52eb7a85f4b3\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") "
Sep 10 23:28:18.941143 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.940894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-token\") pod \"3d952917-3a68-4659-8674-52eb7a85f4b3\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") "
Sep 10 23:28:18.941143 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.940929    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d952917-3a68-4659-8674-52eb7a85f4b3-istiod-ca-cert\") pod \"3d952917-3a68-4659-8674-52eb7a85f4b3\" (UID: \"3d952917-3a68-4659-8674-52eb7a85f4b3\") "
Sep 10 23:28:18.941402 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.941375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d952917-3a68-4659-8674-52eb7a85f4b3" (UID: "3d952917-3a68-4659-8674-52eb7a85f4b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:18.941649 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.941624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d952917-3a68-4659-8674-52eb7a85f4b3" (UID: "3d952917-3a68-4659-8674-52eb7a85f4b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:18.941741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.941656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d952917-3a68-4659-8674-52eb7a85f4b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d952917-3a68-4659-8674-52eb7a85f4b3" (UID: "3d952917-3a68-4659-8674-52eb7a85f4b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:28:18.941806 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.941772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d952917-3a68-4659-8674-52eb7a85f4b3" (UID: "3d952917-3a68-4659-8674-52eb7a85f4b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:18.941889 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.941869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d952917-3a68-4659-8674-52eb7a85f4b3" (UID: "3d952917-3a68-4659-8674-52eb7a85f4b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:18.943854 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.943820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d952917-3a68-4659-8674-52eb7a85f4b3" (UID: "3d952917-3a68-4659-8674-52eb7a85f4b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:18.944329 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.944306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d952917-3a68-4659-8674-52eb7a85f4b3" (UID: "3d952917-3a68-4659-8674-52eb7a85f4b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:28:18.944407 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.944324    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d952917-3a68-4659-8674-52eb7a85f4b3" (UID: "3d952917-3a68-4659-8674-52eb7a85f4b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:18.944407 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:18.944340    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-kube-api-access-t8k6c" (OuterVolumeSpecName: "kube-api-access-t8k6c") pod "3d952917-3a68-4659-8674-52eb7a85f4b3" (UID: "3d952917-3a68-4659-8674-52eb7a85f4b3"). InnerVolumeSpecName "kube-api-access-t8k6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:19.042187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.042094    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:19.042187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.042120    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:19.042187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.042129    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d952917-3a68-4659-8674-52eb7a85f4b3-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:19.042187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.042136    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:19.042187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.042144    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:19.042187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.042151    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:19.042187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.042159    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:19.042187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.042166    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8k6c\" (UniqueName: \"kubernetes.io/projected/3d952917-3a68-4659-8674-52eb7a85f4b3-kube-api-access-t8k6c\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:19.042187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.042174    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d952917-3a68-4659-8674-52eb7a85f4b3-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:19.730960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.730913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7846-cc9769" event={"ID":"a9fb65b0-20b8-44ca-94a8-67a756dfa3a9","Type":"ContainerStarted","Data":"d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5"}
Sep 10 23:28:19.732649 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.732623    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerID="a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b" exitCode=0
Sep 10 23:28:19.732791 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.732665    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"
Sep 10 23:28:19.732791 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.732676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" event={"ID":"3d952917-3a68-4659-8674-52eb7a85f4b3","Type":"ContainerDied","Data":"a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b"}
Sep 10 23:28:19.732791 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.732724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4" event={"ID":"3d952917-3a68-4659-8674-52eb7a85f4b3","Type":"ContainerDied","Data":"074138bd4a6bf4ba973dd9418278bfe988a18d4551f1f9ef7e8eb4876926d91a"}
Sep 10 23:28:19.732791 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.732753    2580 scope.go:117] "RemoveContainer" containerID="a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b"
Sep 10 23:28:19.742772 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.742746    2580 scope.go:117] "RemoveContainer" containerID="6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6"
Sep 10 23:28:19.745897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.745844    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7846-cc9769" podStartSLOduration=2.525916746 podStartE2EDuration="4.745823244s" totalImagesPullingTime="2.219906498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:28:19.743838338 +0000 UTC m=+1599.445471285" watchObservedRunningTime="2026-09-10 23:28:19.745823244 +0000 UTC m=+1599.447456188"
Sep 10 23:28:19.753419 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.753370    2580 scope.go:117] "RemoveContainer" containerID="a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b"
Sep 10 23:28:19.753887 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:28:19.753841    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b\": container with ID starting with a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b not found: ID does not exist" containerID="a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b"
Sep 10 23:28:19.754197 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.754061    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b"} err="failed to get container status \"a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b\": rpc error: code = NotFound desc = could not find container \"a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b\": container with ID starting with a559ee25f12f0f13db9f1cd5d169a63c334c343d861c857ad1d3dcd1630bb84b not found: ID does not exist"
Sep 10 23:28:19.754197 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.754092    2580 scope.go:117] "RemoveContainer" containerID="6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6"
Sep 10 23:28:19.754490 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:28:19.754467    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6\": container with ID starting with 6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6 not found: ID does not exist" containerID="6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6"
Sep 10 23:28:19.754588 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.754499    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6"} err="failed to get container status \"6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6\": rpc error: code = NotFound desc = could not find container \"6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6\": container with ID starting with 6b30bc50759ba66fc169f3278457a2b1e42638da590661848b14f684073cb7d6 not found: ID does not exist"
Sep 10 23:28:19.757043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.757022    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"]
Sep 10 23:28:19.760416 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:19.760397    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7b202f-openshift-default-5687c5cfd-rhzq4"]
Sep 10 23:28:20.320849 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:20.320818    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7846-cc9769"]
Sep 10 23:28:20.325413 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:20.325385    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7846-cc9769"]
Sep 10 23:28:20.739766 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:28:20.739740    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7846-cc9769" container="test-curl-7846-cc9769"
Sep 10 23:28:20.835655 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:20.835615    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" path="/var/lib/kubelet/pods/3d952917-3a68-4659-8674-52eb7a85f4b3/volumes"
Sep 10 23:28:21.675885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:21.675858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:21.751166 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:21.751129    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7846-cc9769" podUID="a9fb65b0-20b8-44ca-94a8-67a756dfa3a9" containerName="test-curl-7846-cc9769" containerID="cri-o://d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5" gracePeriod=2
Sep 10 23:28:22.755181 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:22.755151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:23.928783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:23.928753    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-cc9769"
Sep 10 23:28:23.986716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:23.986662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rfjw\" (UniqueName: \"kubernetes.io/projected/a9fb65b0-20b8-44ca-94a8-67a756dfa3a9-kube-api-access-4rfjw\") pod \"a9fb65b0-20b8-44ca-94a8-67a756dfa3a9\" (UID: \"a9fb65b0-20b8-44ca-94a8-67a756dfa3a9\") "
Sep 10 23:28:23.989049 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:23.989023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fb65b0-20b8-44ca-94a8-67a756dfa3a9-kube-api-access-4rfjw" (OuterVolumeSpecName: "kube-api-access-4rfjw") pod "a9fb65b0-20b8-44ca-94a8-67a756dfa3a9" (UID: "a9fb65b0-20b8-44ca-94a8-67a756dfa3a9"). InnerVolumeSpecName "kube-api-access-4rfjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:24.087926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:24.087840    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rfjw\" (UniqueName: \"kubernetes.io/projected/a9fb65b0-20b8-44ca-94a8-67a756dfa3a9-kube-api-access-4rfjw\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:28:24.777428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:24.777391    2580 generic.go:356] "Generic (PLEG): container finished" podID="a9fb65b0-20b8-44ca-94a8-67a756dfa3a9" containerID="d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5" exitCode=137
Sep 10 23:28:24.777428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:24.777434    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-cc9769"
Sep 10 23:28:24.777690 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:24.777499    2580 scope.go:117] "RemoveContainer" containerID="d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5"
Sep 10 23:28:24.787265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:24.787245    2580 scope.go:117] "RemoveContainer" containerID="d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5"
Sep 10 23:28:24.787329 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:28:24.787256    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7846-cc9769\" is forbidden: User \"system:node:ip-10-0-135-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-96.ec2.internal' and this object" podUID="a9fb65b0-20b8-44ca-94a8-67a756dfa3a9" pod="opendatahub/test-curl-7846-cc9769"
Sep 10 23:28:24.787608 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:28:24.787589    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5\": container with ID starting with d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5 not found: ID does not exist" containerID="d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5"
Sep 10 23:28:24.787668 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:24.787618    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5"} err="failed to get container status \"d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5\": rpc error: code = NotFound desc = could not find container \"d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5\": container with ID starting with d66e3709398a12f7c59d98b377710dc4fe3d0ba52f0cae76e8337fda872402b5 not found: ID does not exist"
Sep 10 23:28:24.834756 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:24.834721    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9fb65b0-20b8-44ca-94a8-67a756dfa3a9" path="/var/lib/kubelet/pods/a9fb65b0-20b8-44ca-94a8-67a756dfa3a9/volumes"
Sep 10 23:28:25.871069 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:25.871007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:26.253498 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.253413    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"]
Sep 10 23:28:26.254035 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254014    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy"
Sep 10 23:28:26.254087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254041    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy"
Sep 10 23:28:26.254087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254058    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy"
Sep 10 23:28:26.254087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254067    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy"
Sep 10 23:28:26.254087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254076    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fb65b0-20b8-44ca-94a8-67a756dfa3a9" containerName="test-curl-7846-cc9769"
Sep 10 23:28:26.254087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254086    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fb65b0-20b8-44ca-94a8-67a756dfa3a9" containerName="test-curl-7846-cc9769"
Sep 10 23:28:26.254253 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254110    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy"
Sep 10 23:28:26.254253 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254119    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy"
Sep 10 23:28:26.254253 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254135    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy"
Sep 10 23:28:26.254253 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254145    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy"
Sep 10 23:28:26.254253 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254243    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eede480-ca51-42f6-8c7b-a21769018b92" containerName="istio-proxy"
Sep 10 23:28:26.254405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254256    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="75184e59-63f8-418f-907a-265e873534dd" containerName="istio-proxy"
Sep 10 23:28:26.254405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254268    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy"
Sep 10 23:28:26.254405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254277    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d952917-3a68-4659-8674-52eb7a85f4b3" containerName="istio-proxy"
Sep 10 23:28:26.254405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254288    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9fb65b0-20b8-44ca-94a8-67a756dfa3a9" containerName="test-curl-7846-cc9769"
Sep 10 23:28:26.254405 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.254296    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25a7d0cd-a7db-4fe3-991c-3e4e5959b43d" containerName="istio-proxy"
Sep 10 23:28:26.259285 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.259268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.261764 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.261731    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"default-dockercfg-vxzdz\""
Sep 10 23:28:26.262430 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.262006    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"openshift-service-ca.crt\""
Sep 10 23:28:26.262430 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.262210    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"kube-root-ca.crt\""
Sep 10 23:28:26.262430 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.262276    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"sub-test-model-6c97f2-kserve-self-signed-certs\""
Sep 10 23:28:26.265691 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.265607    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"]
Sep 10 23:28:26.307836 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.307709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-home\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.307836 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.307795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvpb\" (UniqueName: \"kubernetes.io/projected/462406e8-7d26-493f-a629-4da5df7b4348-kube-api-access-lzvpb\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.308096 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.307855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462406e8-7d26-493f-a629-4da5df7b4348-tls-certs\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.308096 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.307904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-model-cache\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.308096 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.307929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-dshm\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.308096 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.307995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-tmp-dir\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.347855 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.347809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:26.409238 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.409199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvpb\" (UniqueName: \"kubernetes.io/projected/462406e8-7d26-493f-a629-4da5df7b4348-kube-api-access-lzvpb\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.409419 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.409252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462406e8-7d26-493f-a629-4da5df7b4348-tls-certs\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.409419 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.409287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-model-cache\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.409419 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.409319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-dshm\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.409419 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.409355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-tmp-dir\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.409419 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.409389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-home\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.409789 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.409769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-model-cache\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.409848 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.409790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-home\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.409940 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.409924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-tmp-dir\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.411732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.411710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-dshm\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.411963 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.411947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462406e8-7d26-493f-a629-4da5df7b4348-tls-certs\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.419849 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.419819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvpb\" (UniqueName: \"kubernetes.io/projected/462406e8-7d26-493f-a629-4da5df7b4348-kube-api-access-lzvpb\") pod \"sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.572197 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.572089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:26.855095 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:26.855069    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"]
Sep 10 23:28:26.856876 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:28:26.856844    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod462406e8_7d26_493f_a629_4da5df7b4348.slice/crio-c67fe9ddf180e56ee7e89a39ea82adaeaba56ddb6fde7abead2d0a21c1a1acc0 WatchSource:0}: Error finding container c67fe9ddf180e56ee7e89a39ea82adaeaba56ddb6fde7abead2d0a21c1a1acc0: Status 404 returned error can't find the container with id c67fe9ddf180e56ee7e89a39ea82adaeaba56ddb6fde7abead2d0a21c1a1acc0
Sep 10 23:28:27.806448 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:27.806402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x" event={"ID":"462406e8-7d26-493f-a629-4da5df7b4348","Type":"ContainerStarted","Data":"e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59"}
Sep 10 23:28:27.806919 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:27.806459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x" event={"ID":"462406e8-7d26-493f-a629-4da5df7b4348","Type":"ContainerStarted","Data":"c67fe9ddf180e56ee7e89a39ea82adaeaba56ddb6fde7abead2d0a21c1a1acc0"}
Sep 10 23:28:27.823460 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:27.823399    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x" podStartSLOduration=1.82338139 podStartE2EDuration="1.82338139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:28:27.821024489 +0000 UTC m=+1607.522657447" watchObservedRunningTime="2026-09-10 23:28:27.82338139 +0000 UTC m=+1607.525014334"
Sep 10 23:28:36.572290 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:36.572178    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:36.572830 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:36.572471    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:36.585299 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:36.585275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:36.870797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:36.870766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:28:37.876866 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:37.876836    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:28:47.599933 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:47.599896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:47.817067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:47.817036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:52.662481 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:28:52.662448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:03.232124 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:03.232082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:03.278240 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:03.278201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:29:03.310452 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:03.310412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:03.368503 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:03.368474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:03.437240 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:03.437201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:03.492395 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:03.492280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:03.793337 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:03.793233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:09.646179 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:09.646141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:09.746881 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:09.746840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:09.890199 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:09.890163    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"]
Sep 10 23:29:09.890427 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:09.890400    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" containerID="cri-o://3ea29df5541e0588cfbce0197722e009c0c5a04174e20fb5f0a9ba5895afbc42" gracePeriod=30
Sep 10 23:29:09.898690 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:09.898612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:13.777134 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:13.777097    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 23:29:13.777500 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:13.777167    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:29:15.078212 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.078177    2580 generic.go:356] "Generic (PLEG): container finished" podID="78b55da4-111f-4033-841f-5de247dcad01" containerID="3ea29df5541e0588cfbce0197722e009c0c5a04174e20fb5f0a9ba5895afbc42" exitCode=0
Sep 10 23:29:15.078672 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.078245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" event={"ID":"78b55da4-111f-4033-841f-5de247dcad01","Type":"ContainerDied","Data":"3ea29df5541e0588cfbce0197722e009c0c5a04174e20fb5f0a9ba5895afbc42"}
Sep 10 23:29:15.078672 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.078322    2580 scope.go:117] "RemoveContainer" containerID="92a33e98469af3d5e7930aa791d09b95e5c206ab3f9aa7063c73e249c34c469b"
Sep 10 23:29:15.143795 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.143773    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:29:15.282648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282556    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-certs\") pod \"78b55da4-111f-4033-841f-5de247dcad01\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") "
Sep 10 23:29:15.282648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282615    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnsfg\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-kube-api-access-pnsfg\") pod \"78b55da4-111f-4033-841f-5de247dcad01\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") "
Sep 10 23:29:15.282648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-data\") pod \"78b55da4-111f-4033-841f-5de247dcad01\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") "
Sep 10 23:29:15.282935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-envoy\") pod \"78b55da4-111f-4033-841f-5de247dcad01\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") "
Sep 10 23:29:15.282935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282745    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b55da4-111f-4033-841f-5de247dcad01-istiod-ca-cert\") pod \"78b55da4-111f-4033-841f-5de247dcad01\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") "
Sep 10 23:29:15.282935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282776    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b55da4-111f-4033-841f-5de247dcad01-istio-podinfo\") pod \"78b55da4-111f-4033-841f-5de247dcad01\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") "
Sep 10 23:29:15.282935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282807    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-credential-socket\") pod \"78b55da4-111f-4033-841f-5de247dcad01\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") "
Sep 10 23:29:15.282935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282850    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-socket\") pod \"78b55da4-111f-4033-841f-5de247dcad01\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") "
Sep 10 23:29:15.282935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78b55da4-111f-4033-841f-5de247dcad01" (UID: "78b55da4-111f-4033-841f-5de247dcad01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:15.282935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-istio-token\") pod \"78b55da4-111f-4033-841f-5de247dcad01\" (UID: \"78b55da4-111f-4033-841f-5de247dcad01\") "
Sep 10 23:29:15.283263 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.282973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-data" (OuterVolumeSpecName: "istio-data") pod "78b55da4-111f-4033-841f-5de247dcad01" (UID: "78b55da4-111f-4033-841f-5de247dcad01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:15.283263 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.283160    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b55da4-111f-4033-841f-5de247dcad01-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78b55da4-111f-4033-841f-5de247dcad01" (UID: "78b55da4-111f-4033-841f-5de247dcad01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:15.283400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.283293    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:29:15.283454 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.283410    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:29:15.283454 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.283425    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b55da4-111f-4033-841f-5de247dcad01-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:29:15.283581 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.283517    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78b55da4-111f-4033-841f-5de247dcad01" (UID: "78b55da4-111f-4033-841f-5de247dcad01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:15.283785 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.283768    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78b55da4-111f-4033-841f-5de247dcad01" (UID: "78b55da4-111f-4033-841f-5de247dcad01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:15.285130 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.285097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78b55da4-111f-4033-841f-5de247dcad01-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78b55da4-111f-4033-841f-5de247dcad01" (UID: "78b55da4-111f-4033-841f-5de247dcad01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:29:15.285279 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.285262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78b55da4-111f-4033-841f-5de247dcad01" (UID: "78b55da4-111f-4033-841f-5de247dcad01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:15.285562 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.285524    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-istio-token" (OuterVolumeSpecName: "istio-token") pod "78b55da4-111f-4033-841f-5de247dcad01" (UID: "78b55da4-111f-4033-841f-5de247dcad01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:15.285632 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.285558    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-kube-api-access-pnsfg" (OuterVolumeSpecName: "kube-api-access-pnsfg") pod "78b55da4-111f-4033-841f-5de247dcad01" (UID: "78b55da4-111f-4033-841f-5de247dcad01"). InnerVolumeSpecName "kube-api-access-pnsfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:15.384769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.384732    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnsfg\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-kube-api-access-pnsfg\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:29:15.384769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.384763    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:29:15.384769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.384773    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b55da4-111f-4033-841f-5de247dcad01-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:29:15.384769 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.384781    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:29:15.385043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.384790    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b55da4-111f-4033-841f-5de247dcad01-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:29:15.385043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:15.384798    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b55da4-111f-4033-841f-5de247dcad01-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:29:16.085774 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:16.085742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml" event={"ID":"78b55da4-111f-4033-841f-5de247dcad01","Type":"ContainerDied","Data":"b11250dd0a1d09f14537b12845197395c1b10cae42c69c96ebd86abb94cb6462"}
Sep 10 23:29:16.086195 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:16.085798    2580 scope.go:117] "RemoveContainer" containerID="3ea29df5541e0588cfbce0197722e009c0c5a04174e20fb5f0a9ba5895afbc42"
Sep 10 23:29:16.086195 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:16.085758    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"
Sep 10 23:29:16.106672 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:16.106647    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"]
Sep 10 23:29:16.110742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:16.110716    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7025b9-openshift-default-7b6bccd4d6-fbnml"]
Sep 10 23:29:16.834761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:16.834723    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b55da4-111f-4033-841f-5de247dcad01" path="/var/lib/kubelet/pods/78b55da4-111f-4033-841f-5de247dcad01/volumes"
Sep 10 23:29:20.272479 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:20.272441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:29:29.923742 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:29.923698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:30.028971 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:30.028811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:30.478101 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:30.478073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:35.367300 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:35.367262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:38.875639 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:38.875603    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:29:45.272065 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:45.272031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:52.376503 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:29:52.376466    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:30:05.274265 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:05.274161    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:30:09.676621 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.676584    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7846-913552"]
Sep 10 23:30:09.676996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.676978    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy"
Sep 10 23:30:09.676996 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.676990    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy"
Sep 10 23:30:09.677067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.676998    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy"
Sep 10 23:30:09.677067 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.677003    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy"
Sep 10 23:30:09.677133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.677068    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy"
Sep 10 23:30:09.677133 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.677075    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b55da4-111f-4033-841f-5de247dcad01" containerName="istio-proxy"
Sep 10 23:30:09.681506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.681490    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-913552"
Sep 10 23:30:09.683524 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.683495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4p5h\""
Sep 10 23:30:09.686063 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.686039    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7846-913552"]
Sep 10 23:30:09.804707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.804671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bsc\" (UniqueName: \"kubernetes.io/projected/7da32012-8ddc-4856-bbb9-21badd13f3d7-kube-api-access-s2bsc\") pod \"test-curl-7846-913552\" (UID: \"7da32012-8ddc-4856-bbb9-21badd13f3d7\") " pod="opendatahub/test-curl-7846-913552"
Sep 10 23:30:09.906213 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.906172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bsc\" (UniqueName: \"kubernetes.io/projected/7da32012-8ddc-4856-bbb9-21badd13f3d7-kube-api-access-s2bsc\") pod \"test-curl-7846-913552\" (UID: \"7da32012-8ddc-4856-bbb9-21badd13f3d7\") " pod="opendatahub/test-curl-7846-913552"
Sep 10 23:30:09.913917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.913897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bsc\" (UniqueName: \"kubernetes.io/projected/7da32012-8ddc-4856-bbb9-21badd13f3d7-kube-api-access-s2bsc\") pod \"test-curl-7846-913552\" (UID: \"7da32012-8ddc-4856-bbb9-21badd13f3d7\") " pod="opendatahub/test-curl-7846-913552"
Sep 10 23:30:09.993363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:09.993280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-913552"
Sep 10 23:30:10.262960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:10.262932    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7846-913552"]
Sep 10 23:30:10.264632 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:30:10.264605    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7da32012_8ddc_4856_bbb9_21badd13f3d7.slice/crio-4afb02aafa8393fcfc24ad5617d9567fd477fe38357aa3e26f962e1d5f75ea4d WatchSource:0}: Error finding container 4afb02aafa8393fcfc24ad5617d9567fd477fe38357aa3e26f962e1d5f75ea4d: Status 404 returned error can't find the container with id 4afb02aafa8393fcfc24ad5617d9567fd477fe38357aa3e26f962e1d5f75ea4d
Sep 10 23:30:10.414034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:10.413997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7846-913552" event={"ID":"7da32012-8ddc-4856-bbb9-21badd13f3d7","Type":"ContainerStarted","Data":"ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d"}
Sep 10 23:30:10.414206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:10.414043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7846-913552" event={"ID":"7da32012-8ddc-4856-bbb9-21badd13f3d7","Type":"ContainerStarted","Data":"4afb02aafa8393fcfc24ad5617d9567fd477fe38357aa3e26f962e1d5f75ea4d"}
Sep 10 23:30:10.428796 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:10.428742    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7846-913552" podStartSLOduration=1.428724954 podStartE2EDuration="1.428724954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:30:10.426588466 +0000 UTC m=+1710.128221405" watchObservedRunningTime="2026-09-10 23:30:10.428724954 +0000 UTC m=+1710.130357897"
Sep 10 23:30:11.000958 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:11.000921    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7846-913552"]
Sep 10 23:30:11.004418 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:11.004393    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7846-913552"]
Sep 10 23:30:11.419235 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:30:11.419206    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7846-913552" container="test-curl-7846-913552"
Sep 10 23:30:12.424827 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:12.424786    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7846-913552" podUID="7da32012-8ddc-4856-bbb9-21badd13f3d7" containerName="test-curl-7846-913552" containerID="cri-o://ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d" gracePeriod=2
Sep 10 23:30:12.426752 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:30:12.426723    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7846-913552\" is forbidden: User \"system:node:ip-10-0-135-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-96.ec2.internal' and this object" podUID="7da32012-8ddc-4856-bbb9-21badd13f3d7" pod="opendatahub/test-curl-7846-913552"
Sep 10 23:30:12.929351 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:12.929320    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"]
Sep 10 23:30:12.945985 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:12.945945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:30:13.981671 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:13.981641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:30:13.983367 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:13.983342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:30:14.607173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:14.607147    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-913552"
Sep 10 23:30:14.758811 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:14.758716    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2bsc\" (UniqueName: \"kubernetes.io/projected/7da32012-8ddc-4856-bbb9-21badd13f3d7-kube-api-access-s2bsc\") pod \"7da32012-8ddc-4856-bbb9-21badd13f3d7\" (UID: \"7da32012-8ddc-4856-bbb9-21badd13f3d7\") "
Sep 10 23:30:14.760992 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:14.760961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da32012-8ddc-4856-bbb9-21badd13f3d7-kube-api-access-s2bsc" (OuterVolumeSpecName: "kube-api-access-s2bsc") pod "7da32012-8ddc-4856-bbb9-21badd13f3d7" (UID: "7da32012-8ddc-4856-bbb9-21badd13f3d7"). InnerVolumeSpecName "kube-api-access-s2bsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:30:14.833783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:14.833740    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7da32012-8ddc-4856-bbb9-21badd13f3d7" path="/var/lib/kubelet/pods/7da32012-8ddc-4856-bbb9-21badd13f3d7/volumes"
Sep 10 23:30:14.859712 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:14.859685    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2bsc\" (UniqueName: \"kubernetes.io/projected/7da32012-8ddc-4856-bbb9-21badd13f3d7-kube-api-access-s2bsc\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:30:15.443148 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:15.443115    2580 generic.go:356] "Generic (PLEG): container finished" podID="7da32012-8ddc-4856-bbb9-21badd13f3d7" containerID="ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d" exitCode=137
Sep 10 23:30:15.443604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:15.443160    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-913552"
Sep 10 23:30:15.443604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:15.443218    2580 scope.go:117] "RemoveContainer" containerID="ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d"
Sep 10 23:30:15.452045 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:15.452025    2580 scope.go:117] "RemoveContainer" containerID="ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d"
Sep 10 23:30:15.452314 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:30:15.452295    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d\": container with ID starting with ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d not found: ID does not exist" containerID="ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d"
Sep 10 23:30:15.452400 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:15.452322    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d"} err="failed to get container status \"ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d\": rpc error: code = NotFound desc = could not find container \"ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d\": container with ID starting with ab303b15d2cb3847714297169aca316e5e015c57e4be055aa1597973ffbb890d not found: ID does not exist"
Sep 10 23:30:22.177782 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:22.177746    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:30:27.949534 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:27.949486    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x" podUID="462406e8-7d26-493f-a629-4da5df7b4348" containerName="main" containerID="cri-o://e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59" gracePeriod=45
Sep 10 23:30:28.201727 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.201656    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:30:28.279974 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.279939    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-model-cache\") pod \"462406e8-7d26-493f-a629-4da5df7b4348\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") "
Sep 10 23:30:28.280180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280001    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462406e8-7d26-493f-a629-4da5df7b4348-tls-certs\") pod \"462406e8-7d26-493f-a629-4da5df7b4348\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") "
Sep 10 23:30:28.280180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280058    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-tmp-dir\") pod \"462406e8-7d26-493f-a629-4da5df7b4348\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") "
Sep 10 23:30:28.280180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280129    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-home\") pod \"462406e8-7d26-493f-a629-4da5df7b4348\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") "
Sep 10 23:30:28.280180 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-dshm\") pod \"462406e8-7d26-493f-a629-4da5df7b4348\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") "
Sep 10 23:30:28.280392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280198    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzvpb\" (UniqueName: \"kubernetes.io/projected/462406e8-7d26-493f-a629-4da5df7b4348-kube-api-access-lzvpb\") pod \"462406e8-7d26-493f-a629-4da5df7b4348\" (UID: \"462406e8-7d26-493f-a629-4da5df7b4348\") "
Sep 10 23:30:28.280392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280227    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-model-cache" (OuterVolumeSpecName: "model-cache") pod "462406e8-7d26-493f-a629-4da5df7b4348" (UID: "462406e8-7d26-493f-a629-4da5df7b4348"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:30:28.280392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280290    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "462406e8-7d26-493f-a629-4da5df7b4348" (UID: "462406e8-7d26-493f-a629-4da5df7b4348"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:30:28.280527 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280409    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-home" (OuterVolumeSpecName: "home") pod "462406e8-7d26-493f-a629-4da5df7b4348" (UID: "462406e8-7d26-493f-a629-4da5df7b4348"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:30:28.280626 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280611    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-home\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:30:28.280674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280628    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-model-cache\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:30:28.280674 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.280637    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-tmp-dir\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:30:28.282446 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.282423    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462406e8-7d26-493f-a629-4da5df7b4348-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "462406e8-7d26-493f-a629-4da5df7b4348" (UID: "462406e8-7d26-493f-a629-4da5df7b4348"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:30:28.282841 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.282824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-dshm" (OuterVolumeSpecName: "dshm") pod "462406e8-7d26-493f-a629-4da5df7b4348" (UID: "462406e8-7d26-493f-a629-4da5df7b4348"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:30:28.282897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.282863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462406e8-7d26-493f-a629-4da5df7b4348-kube-api-access-lzvpb" (OuterVolumeSpecName: "kube-api-access-lzvpb") pod "462406e8-7d26-493f-a629-4da5df7b4348" (UID: "462406e8-7d26-493f-a629-4da5df7b4348"). InnerVolumeSpecName "kube-api-access-lzvpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:30:28.381667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.381629    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462406e8-7d26-493f-a629-4da5df7b4348-dshm\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:30:28.381667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.381663    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzvpb\" (UniqueName: \"kubernetes.io/projected/462406e8-7d26-493f-a629-4da5df7b4348-kube-api-access-lzvpb\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:30:28.381667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.381676    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462406e8-7d26-493f-a629-4da5df7b4348-tls-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:30:28.518061 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.517970    2580 generic.go:356] "Generic (PLEG): container finished" podID="462406e8-7d26-493f-a629-4da5df7b4348" containerID="e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59" exitCode=0
Sep 10 23:30:28.518061 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.518014    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"
Sep 10 23:30:28.518315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.518064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x" event={"ID":"462406e8-7d26-493f-a629-4da5df7b4348","Type":"ContainerDied","Data":"e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59"}
Sep 10 23:30:28.518315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.518112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x" event={"ID":"462406e8-7d26-493f-a629-4da5df7b4348","Type":"ContainerDied","Data":"c67fe9ddf180e56ee7e89a39ea82adaeaba56ddb6fde7abead2d0a21c1a1acc0"}
Sep 10 23:30:28.518315 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.518137    2580 scope.go:117] "RemoveContainer" containerID="e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59"
Sep 10 23:30:28.528087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.528069    2580 scope.go:117] "RemoveContainer" containerID="e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59"
Sep 10 23:30:28.528367 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:30:28.528345    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59\": container with ID starting with e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59 not found: ID does not exist" containerID="e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59"
Sep 10 23:30:28.528413 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.528376    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59"} err="failed to get container status \"e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59\": rpc error: code = NotFound desc = could not find container \"e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59\": container with ID starting with e06090cf5c831a1e4281dfa901c5464f409b519eb5c58977cfe8dd973bbd8a59 not found: ID does not exist"
Sep 10 23:30:28.538863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.538838    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"]
Sep 10 23:30:28.542908 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.542885    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/sub-test-model-6c97f2-kserve-5f56cffb49-9cf4x"]
Sep 10 23:30:28.834557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:28.834467    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462406e8-7d26-493f-a629-4da5df7b4348" path="/var/lib/kubelet/pods/462406e8-7d26-493f-a629-4da5df7b4348/volumes"
Sep 10 23:30:32.377926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:32.377895    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:30:35.476069 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:35.476030    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:30:41.391411 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:30:41.391377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:05.379847 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:05.379813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:10.280335 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:10.280291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:14.074118 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:14.074085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:31:20.933098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:20.933044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:20.988424 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:20.988393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:24.296130 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:24.296098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:26.977955 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:26.977917    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:31:36.773999 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:36.773914    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:31:42.850438 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:42.850364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:43.785665 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:43.785633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:44.215341 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.215307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:44.498745 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.498663    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"]
Sep 10 23:31:44.499170 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.499151    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462406e8-7d26-493f-a629-4da5df7b4348" containerName="main"
Sep 10 23:31:44.499249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.499174    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="462406e8-7d26-493f-a629-4da5df7b4348" containerName="main"
Sep 10 23:31:44.499249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.499202    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da32012-8ddc-4856-bbb9-21badd13f3d7" containerName="test-curl-7846-913552"
Sep 10 23:31:44.499249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.499209    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da32012-8ddc-4856-bbb9-21badd13f3d7" containerName="test-curl-7846-913552"
Sep 10 23:31:44.499360 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.499278    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da32012-8ddc-4856-bbb9-21badd13f3d7" containerName="test-curl-7846-913552"
Sep 10 23:31:44.499360 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.499294    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="462406e8-7d26-493f-a629-4da5df7b4348" containerName="main"
Sep 10 23:31:44.502455 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.502434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.504483 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.504458    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ea2b31-openshift-default-dockercfg-8tbgd\""
Sep 10 23:31:44.512829 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.512805    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"]
Sep 10 23:31:44.513407 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.513380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-certs\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.513509 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.513425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-socket\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.513509 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.513451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s458c\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-kube-api-access-s458c\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.513509 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.513477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-token\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.513714 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.513632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istiod-ca-cert\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.513714 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.513679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-podinfo\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.513814 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.513711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-credential-socket\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.513814 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.513775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-envoy\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.513896 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.513839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-data\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.524998 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.524971    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:44.615036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.614995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istiod-ca-cert\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.615242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-podinfo\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.615242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-credential-socket\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.615242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-envoy\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.615242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-data\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.615242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-certs\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.615242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-socket\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.615568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s458c\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-kube-api-access-s458c\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.615568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-token\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.615685 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615587    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-credential-socket\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.616016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-certs\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.616016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istiod-ca-cert\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.616016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.615961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-socket\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.616016 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.616015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-data\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.618758 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.618732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-podinfo\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.621564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.621320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-envoy\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.624443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.624398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-token\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.624858 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.624827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s458c\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-kube-api-access-s458c\") pod \"e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") " pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.814620 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.814513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:44.939351 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:44.939310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:45.090966 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:45.090942    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"]
Sep 10 23:31:45.093000 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:31:45.092973    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4144c225_7fe6_4e05_a1c9_9bc34b85e045.slice/crio-98a541b29e4ee98d2565dcd1e4395f18f608b424bd4a94aa7345aa6ac58aef89 WatchSource:0}: Error finding container 98a541b29e4ee98d2565dcd1e4395f18f608b424bd4a94aa7345aa6ac58aef89: Status 404 returned error can't find the container with id 98a541b29e4ee98d2565dcd1e4395f18f608b424bd4a94aa7345aa6ac58aef89
Sep 10 23:31:45.095234 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:45.095194    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:31:45.095305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:45.095289    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:31:45.095346 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:45.095335    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:31:45.956823 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:45.956787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" event={"ID":"4144c225-7fe6-4e05-a1c9-9bc34b85e045","Type":"ContainerStarted","Data":"5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065"}
Sep 10 23:31:45.957199 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:45.956849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" event={"ID":"4144c225-7fe6-4e05-a1c9-9bc34b85e045","Type":"ContainerStarted","Data":"98a541b29e4ee98d2565dcd1e4395f18f608b424bd4a94aa7345aa6ac58aef89"}
Sep 10 23:31:45.976078 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:45.976025    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" podStartSLOduration=1.976008647 podStartE2EDuration="1.976008647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:31:45.973516572 +0000 UTC m=+1805.675149530" watchObservedRunningTime="2026-09-10 23:31:45.976008647 +0000 UTC m=+1805.677641589"
Sep 10 23:31:46.815412 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:46.815377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:46.820758 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:46.820735    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:46.963436 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:46.963407    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:46.964439 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:46.964422    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:31:47.375033 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:47.374994    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:31:47.860041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:47.860011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:47.961841 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:47.961802    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:49.184242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:49.184207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:49.294730 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:49.294640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:49.592909 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:49.592756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:49.693420 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:49.693387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:49.780866 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:49.780830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:59.868191 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:31:59.868157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:32:02.746981 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:02.746806    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": read tcp 10.134.0.2:55604->10.134.0.87:15021: read: connection reset by peer" start-of-body=
Sep 10 23:32:02.746981 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:02.746880    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": read tcp 10.134.0.2:55604->10.134.0.87:15021: read: connection reset by peer"
Sep 10 23:32:03.075229 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:03.075129    2580 generic.go:356] "Generic (PLEG): container finished" podID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerID="5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065" exitCode=0
Sep 10 23:32:03.075229 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:03.075209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" event={"ID":"4144c225-7fe6-4e05-a1c9-9bc34b85e045","Type":"ContainerDied","Data":"5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065"}
Sep 10 23:32:03.075655 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:03.075641    2580 scope.go:117] "RemoveContainer" containerID="5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065"
Sep 10 23:32:03.076790 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:03.076759    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:32:03.076904 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:03.076844    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:32:03.076904 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:03.076889    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:32:04.083430 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:04.083393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" event={"ID":"4144c225-7fe6-4e05-a1c9-9bc34b85e045","Type":"ContainerStarted","Data":"a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56"}
Sep 10 23:32:04.815083 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:04.815056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:32:04.815276 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:04.815244    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 23:32:04.815329 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:04.815301    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 23:32:05.815929 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:05.815895    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 23:32:05.816522 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:05.816490    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 23:32:06.815461 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:06.815427    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 23:32:06.815753 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:06.815512    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 23:32:07.815526 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:07.815491    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 23:32:07.816206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:07.815581    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 23:32:08.815852 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:08.815806    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 23:32:08.816395 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:08.815889    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 23:32:09.818249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:09.818221    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:32:09.818707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:09.818695    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:32:09.819327 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:09.819307    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:32:12.845893 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:12.845856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:13.489794 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.489759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:13.763763 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.763688    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"]
Sep 10 23:32:13.767672 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.767650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:13.770116 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.770094    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-706b9d-openshift-default-dockercfg-b8pwk\""
Sep 10 23:32:13.777690 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.777665    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"]
Sep 10 23:32:13.786572 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.786521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:13.886200 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.886164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:13.913470 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.913441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-credential-socket\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:13.913667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.913498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-token\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:13.913667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.913604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt89r\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-kube-api-access-dt89r\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:13.913798 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.913688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67d0caa-3b8f-4700-b723-2eef33beb664-istiod-ca-cert\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:13.913798 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.913724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-socket\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:13.913798 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.913761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-certs\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:13.913798 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.913789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-podinfo\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:13.913935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.913842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-data\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:13.913935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:13.913898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-envoy\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.014855 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.014763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-credential-socket\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.014855 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.014813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-token\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.014855 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.014844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt89r\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-kube-api-access-dt89r\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.014892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67d0caa-3b8f-4700-b723-2eef33beb664-istiod-ca-cert\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.014916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-socket\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.014937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-certs\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.014961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-podinfo\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.014987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-data\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015159 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.015032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-envoy\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.015268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-credential-socket\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.015345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-socket\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015480 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.015398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-data\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.015645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67d0caa-3b8f-4700-b723-2eef33beb664-istiod-ca-cert\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.015679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.015660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-certs\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.017450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.017422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-envoy\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.017603 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.017581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-podinfo\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.021770 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.021748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-token\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.022089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.022072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt89r\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-kube-api-access-dt89r\") pod \"e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") " pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.081884 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.081856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:14.362590 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.362566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"]
Sep 10 23:32:14.364608 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:32:14.364578    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67d0caa_3b8f_4700_b723_2eef33beb664.slice/crio-058e01de94305e431b802b1003e60ccd905c63b1f2bea97ca2fab2e01a04beb4 WatchSource:0}: Error finding container 058e01de94305e431b802b1003e60ccd905c63b1f2bea97ca2fab2e01a04beb4: Status 404 returned error can't find the container with id 058e01de94305e431b802b1003e60ccd905c63b1f2bea97ca2fab2e01a04beb4
Sep 10 23:32:14.367126 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.367090    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:32:14.367213 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.367160    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:32:14.367213 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:14.367188    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:32:15.155861 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:15.155825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" event={"ID":"d67d0caa-3b8f-4700-b723-2eef33beb664","Type":"ContainerStarted","Data":"13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe"}
Sep 10 23:32:15.156244 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:15.155875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" event={"ID":"d67d0caa-3b8f-4700-b723-2eef33beb664","Type":"ContainerStarted","Data":"058e01de94305e431b802b1003e60ccd905c63b1f2bea97ca2fab2e01a04beb4"}
Sep 10 23:32:15.177927 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:15.177867    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" podStartSLOduration=2.177850193 podStartE2EDuration="2.177850193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:32:15.175321638 +0000 UTC m=+1834.876954580" watchObservedRunningTime="2026-09-10 23:32:15.177850193 +0000 UTC m=+1834.879483136"
Sep 10 23:32:16.082288 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:16.082252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:16.087392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:16.087364    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:16.161834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:16.161803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:16.162839 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:16.162822    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:17.051736 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:17.051672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:17.151660 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:17.151621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:18.289317 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:18.289275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:18.348412 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:18.348343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:18.567972 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:18.567871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:18.796814 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:18.796779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:18.889418 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:18.889388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:24.379665 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:24.379630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:32:35.280585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:35.280534    2580 generic.go:356] "Generic (PLEG): container finished" podID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerID="13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe" exitCode=0
Sep 10 23:32:35.280585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:35.280570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" event={"ID":"d67d0caa-3b8f-4700-b723-2eef33beb664","Type":"ContainerDied","Data":"13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe"}
Sep 10 23:32:35.281091 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:35.281017    2580 scope.go:117] "RemoveContainer" containerID="13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe"
Sep 10 23:32:35.282125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:35.282096    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:32:35.282179 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:35.282165    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:32:35.282220 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:35.282197    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:32:36.288430 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:36.288395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" event={"ID":"d67d0caa-3b8f-4700-b723-2eef33beb664","Type":"ContainerStarted","Data":"ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24"}
Sep 10 23:32:37.082098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:37.082047    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:37.082345 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:37.082274    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Sep 10 23:32:37.082345 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:37.082327    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 10 23:32:38.082582 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:38.082522    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Sep 10 23:32:38.082984 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:38.082624    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 10 23:32:39.082777 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:39.082741    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Sep 10 23:32:39.083149 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:39.082799    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 10 23:32:40.082912 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:40.082867    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Sep 10 23:32:40.083310 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:40.082945    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 10 23:32:41.082242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:41.082210    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Sep 10 23:32:41.082432 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:41.082264    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 10 23:32:42.040365 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:42.040324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:42.086732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:42.086699    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:42.087119 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:42.087098    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:42.087790 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:42.087776    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:32:52.070865 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:52.070832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:32:56.073868 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:56.073835    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:32:57.806521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:32:57.806483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:05.347450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:05.347348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:11.373632 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:11.373599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:33:18.140174 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.140145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:18.417413 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.417306    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"]
Sep 10 23:33:18.419877 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.419853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.422036 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.422013    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-230464-openshift-default-dockercfg-s5lfm\""
Sep 10 23:33:18.432167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.432136    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"]
Sep 10 23:33:18.441233 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.441208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:18.562960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.562927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-data\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.562960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.562961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-token\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.563173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.563006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-podinfo\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.563173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.563034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-certs\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.563173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.563066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-credential-socket\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.563173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.563095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8d8g\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-kube-api-access-z8d8g\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.563173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.563125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-envoy\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.563173 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.563164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-socket\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.563371 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.563179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istiod-ca-cert\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.663830 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.663796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-envoy\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.663860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-socket\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.663889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istiod-ca-cert\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.663938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-data\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.663963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-token\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664034 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.664023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-podinfo\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664309 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.664068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-certs\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664309 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.664110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-credential-socket\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664309 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.664155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8d8g\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-kube-api-access-z8d8g\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664467 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.664370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-socket\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664467 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.664381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-data\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664679 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.664656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-credential-socket\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664763 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.664673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-certs\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.664905 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.664878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istiod-ca-cert\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.666632 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.666613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-envoy\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.667000 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.666983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-podinfo\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.672095 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.672041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8d8g\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-kube-api-access-z8d8g\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.672523 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.672501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-token\") pod \"e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") " pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:18.690404 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.690372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:18.733681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:18.733652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:19.024046 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:19.024014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"]
Sep 10 23:33:19.025794 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:33:19.025750    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c8ef3a_6e3e_4178_ad3f_ccd093c72903.slice/crio-1e4e07ebd11350ea832c4bf95e35fac47e3ee5ac1bcc96ba9754a878a3db4fa6 WatchSource:0}: Error finding container 1e4e07ebd11350ea832c4bf95e35fac47e3ee5ac1bcc96ba9754a878a3db4fa6: Status 404 returned error can't find the container with id 1e4e07ebd11350ea832c4bf95e35fac47e3ee5ac1bcc96ba9754a878a3db4fa6
Sep 10 23:33:19.028748 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:19.028705    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:33:19.028859 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:19.028791    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:33:19.028859 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:19.028829    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:33:19.556357 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:19.556320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" event={"ID":"19c8ef3a-6e3e-4178-ad3f-ccd093c72903","Type":"ContainerStarted","Data":"0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da"}
Sep 10 23:33:19.556865 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:19.556382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" event={"ID":"19c8ef3a-6e3e-4178-ad3f-ccd093c72903","Type":"ContainerStarted","Data":"1e4e07ebd11350ea832c4bf95e35fac47e3ee5ac1bcc96ba9754a878a3db4fa6"}
Sep 10 23:33:19.575741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:19.575688    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" podStartSLOduration=1.5756717569999998 podStartE2EDuration="1.575671757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:33:19.572641585 +0000 UTC m=+1899.274274528" watchObservedRunningTime="2026-09-10 23:33:19.575671757 +0000 UTC m=+1899.277304700"
Sep 10 23:33:19.734661 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:19.734627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:20.739762 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:20.739731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:21.433986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:21.433942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:21.534002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:21.533971    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:21.568942 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:21.568907    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:21.569911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:21.569893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:22.639533 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.639488    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"]
Sep 10 23:33:22.642179 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.642155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.644252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.644230    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-230464-serving-cert\""
Sep 10 23:33:22.655218 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.655194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"]
Sep 10 23:33:22.674761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.674704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:22.774350 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.774321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:22.806975 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.806942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.807143 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.806986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be8ad8-6a4a-418f-8628-4b14814384fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.807143 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.807051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.807143 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.807131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhxq\" (UniqueName: \"kubernetes.io/projected/f7be8ad8-6a4a-418f-8628-4b14814384fc-kube-api-access-vhhxq\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.830088 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.830054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:22.908673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.908529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.908673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.908623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be8ad8-6a4a-418f-8628-4b14814384fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.908926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.908697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.908926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.908782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhxq\" (UniqueName: \"kubernetes.io/projected/f7be8ad8-6a4a-418f-8628-4b14814384fc-kube-api-access-vhhxq\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.909423 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.909395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be8ad8-6a4a-418f-8628-4b14814384fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.911277 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.911256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.911398 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.911381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.916583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.916559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhxq\" (UniqueName: \"kubernetes.io/projected/f7be8ad8-6a4a-418f-8628-4b14814384fc-kube-api-access-vhhxq\") pod \"maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:22.954982 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:22.954951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:23.230078 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:23.229843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:23.231791 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:23.231761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"]
Sep 10 23:33:23.233135 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:33:23.233110    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7be8ad8_6a4a_418f_8628_4b14814384fc.slice/crio-4f6263cea6eb8e64585ba08f82b414b7fbf4175dd668a3568afbee93b26057bf WatchSource:0}: Error finding container 4f6263cea6eb8e64585ba08f82b414b7fbf4175dd668a3568afbee93b26057bf: Status 404 returned error can't find the container with id 4f6263cea6eb8e64585ba08f82b414b7fbf4175dd668a3568afbee93b26057bf
Sep 10 23:33:23.234623 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:23.234601    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:33:23.331081 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:23.331042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:23.584809 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:23.584722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7" event={"ID":"f7be8ad8-6a4a-418f-8628-4b14814384fc","Type":"ContainerStarted","Data":"4f6263cea6eb8e64585ba08f82b414b7fbf4175dd668a3568afbee93b26057bf"}
Sep 10 23:33:24.592925 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:24.592882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7" event={"ID":"f7be8ad8-6a4a-418f-8628-4b14814384fc","Type":"ContainerStarted","Data":"76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c"}
Sep 10 23:33:24.593326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:24.593034    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:24.610601 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:24.610555    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7" podStartSLOduration=2.110001884 podStartE2EDuration="2.610524582s" totalImagesPullingTime="500.522698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:33:24.608965914 +0000 UTC m=+1904.310598857" watchObservedRunningTime="2026-09-10 23:33:24.610524582 +0000 UTC m=+1904.312157526"
Sep 10 23:33:26.541348 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:26.541311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:30.582338 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:30.582301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:33:30.604042 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:30.604015    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:40.700471 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:40.700391    2580 generic.go:356] "Generic (PLEG): container finished" podID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerID="0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da" exitCode=0
Sep 10 23:33:40.700874 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:40.700470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" event={"ID":"19c8ef3a-6e3e-4178-ad3f-ccd093c72903","Type":"ContainerDied","Data":"0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da"}
Sep 10 23:33:40.700918 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:40.700899    2580 scope.go:117] "RemoveContainer" containerID="0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da"
Sep 10 23:33:40.702157 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:40.702129    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:33:40.702208 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:40.702199    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:33:40.702246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:40.702231    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 23:33:41.708313 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:41.708275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" event={"ID":"19c8ef3a-6e3e-4178-ad3f-ccd093c72903","Type":"ContainerStarted","Data":"a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a"}
Sep 10 23:33:41.734143 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:41.734097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:41.734414 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:41.734376    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Sep 10 23:33:41.734499 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:41.734444    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Sep 10 23:33:42.734872 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:42.734832    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Sep 10 23:33:42.736424 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:42.736389    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Sep 10 23:33:43.602361 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:43.602330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:43.734762 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:43.734715    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Sep 10 23:33:43.734970 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:43.734788    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Sep 10 23:33:44.737814 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:44.737772    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Sep 10 23:33:44.738317 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:44.737838    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Sep 10 23:33:45.734428 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:45.734372    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Sep 10 23:33:45.734673 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:45.734455    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Sep 10 23:33:46.440946 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:46.440890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:46.734441 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:46.734329    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Sep 10 23:33:46.734733 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:46.734701    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Sep 10 23:33:47.739196 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:47.739165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:47.739655 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:47.739575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:47.740165 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:47.740146    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:33:53.116027 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.115996    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"]
Sep 10 23:33:53.116449 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.116212    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7" podUID="f7be8ad8-6a4a-418f-8628-4b14814384fc" containerName="maas-api" containerID="cri-o://76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c" gracePeriod=30
Sep 10 23:33:53.135134 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.135100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:53.195606 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.195569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:53.252064 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.252036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:53.317198 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.317157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:53.376511 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.376454    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:53.401897 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.401861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:53.425508 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.425473    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-maas-api-tls\") pod \"f7be8ad8-6a4a-418f-8628-4b14814384fc\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") "
Sep 10 23:33:53.426258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.425764    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhhxq\" (UniqueName: \"kubernetes.io/projected/f7be8ad8-6a4a-418f-8628-4b14814384fc-kube-api-access-vhhxq\") pod \"f7be8ad8-6a4a-418f-8628-4b14814384fc\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") "
Sep 10 23:33:53.426258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.425829    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-metrics-certs\") pod \"f7be8ad8-6a4a-418f-8628-4b14814384fc\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") "
Sep 10 23:33:53.426258 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.425963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be8ad8-6a4a-418f-8628-4b14814384fc-openshift-service-ca\") pod \"f7be8ad8-6a4a-418f-8628-4b14814384fc\" (UID: \"f7be8ad8-6a4a-418f-8628-4b14814384fc\") "
Sep 10 23:33:53.427164 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.427131    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7be8ad8-6a4a-418f-8628-4b14814384fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7be8ad8-6a4a-418f-8628-4b14814384fc" (UID: "f7be8ad8-6a4a-418f-8628-4b14814384fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:33:53.432858 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.432805    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7be8ad8-6a4a-418f-8628-4b14814384fc-kube-api-access-vhhxq" (OuterVolumeSpecName: "kube-api-access-vhhxq") pod "f7be8ad8-6a4a-418f-8628-4b14814384fc" (UID: "f7be8ad8-6a4a-418f-8628-4b14814384fc"). InnerVolumeSpecName "kube-api-access-vhhxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:33:53.433164 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.433142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7be8ad8-6a4a-418f-8628-4b14814384fc" (UID: "f7be8ad8-6a4a-418f-8628-4b14814384fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:33:53.433766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.433743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7be8ad8-6a4a-418f-8628-4b14814384fc" (UID: "f7be8ad8-6a4a-418f-8628-4b14814384fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:33:53.527894 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.527861    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:33:53.527894 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.527894    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be8ad8-6a4a-418f-8628-4b14814384fc-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:33:53.528098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.527905    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be8ad8-6a4a-418f-8628-4b14814384fc-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:33:53.528098 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.527915    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhxq\" (UniqueName: \"kubernetes.io/projected/f7be8ad8-6a4a-418f-8628-4b14814384fc-kube-api-access-vhhxq\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:33:53.669560 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.669450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:53.791106 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.791069    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7be8ad8-6a4a-418f-8628-4b14814384fc" containerID="76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c" exitCode=0
Sep 10 23:33:53.791288 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.791118    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"
Sep 10 23:33:53.791288 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.791148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7" event={"ID":"f7be8ad8-6a4a-418f-8628-4b14814384fc","Type":"ContainerDied","Data":"76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c"}
Sep 10 23:33:53.791288 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.791220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7" event={"ID":"f7be8ad8-6a4a-418f-8628-4b14814384fc","Type":"ContainerDied","Data":"4f6263cea6eb8e64585ba08f82b414b7fbf4175dd668a3568afbee93b26057bf"}
Sep 10 23:33:53.791288 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.791242    2580 scope.go:117] "RemoveContainer" containerID="76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c"
Sep 10 23:33:53.801332 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.801316    2580 scope.go:117] "RemoveContainer" containerID="76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c"
Sep 10 23:33:53.801619 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:33:53.801597    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c\": container with ID starting with 76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c not found: ID does not exist" containerID="76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c"
Sep 10 23:33:53.801701 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.801628    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c"} err="failed to get container status \"76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c\": rpc error: code = NotFound desc = could not find container \"76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c\": container with ID starting with 76d3a311930b662788a7abb52bbbd5c294b2d21367a00494140eef6d4cf4a37c not found: ID does not exist"
Sep 10 23:33:53.811693 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.811674    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"]
Sep 10 23:33:53.815040 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:53.815017    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-230464-7ffd98ccf5-kj2v7"]
Sep 10 23:33:54.834278 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:54.834244    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7be8ad8-6a4a-418f-8628-4b14814384fc" path="/var/lib/kubelet/pods/f7be8ad8-6a4a-418f-8628-4b14814384fc/volumes"
Sep 10 23:33:57.469832 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:57.469793    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:33:59.913174 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:59.913136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:33:59.967431 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:33:59.967397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:00.158987 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:00.158956    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"]
Sep 10 23:34:00.159241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:00.159217    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy" containerID="cri-o://a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a" gracePeriod=30
Sep 10 23:34:00.179247 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:00.179142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:02.741607 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:02.741573    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 10 23:34:02.741988 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:02.741638    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:34:03.777303 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:03.777270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:05.423968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.423903    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:34:05.553589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.553517    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-podinfo\") pod \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") "
Sep 10 23:34:05.553771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.553605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-socket\") pod \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") "
Sep 10 23:34:05.553771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.553640    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-token\") pod \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") "
Sep 10 23:34:05.553771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.553664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-envoy\") pod \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") "
Sep 10 23:34:05.553771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.553720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istiod-ca-cert\") pod \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") "
Sep 10 23:34:05.553771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.553754    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-certs\") pod \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") "
Sep 10 23:34:05.554032 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.553803    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-credential-socket\") pod \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") "
Sep 10 23:34:05.554032 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.553888    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8d8g\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-kube-api-access-z8d8g\") pod \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") "
Sep 10 23:34:05.554032 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.553937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-data\") pod \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\" (UID: \"19c8ef3a-6e3e-4178-ad3f-ccd093c72903\") "
Sep 10 23:34:05.554032 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.554005    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19c8ef3a-6e3e-4178-ad3f-ccd093c72903" (UID: "19c8ef3a-6e3e-4178-ad3f-ccd093c72903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:34:05.554241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.554048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19c8ef3a-6e3e-4178-ad3f-ccd093c72903" (UID: "19c8ef3a-6e3e-4178-ad3f-ccd093c72903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:34:05.554241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.554199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19c8ef3a-6e3e-4178-ad3f-ccd093c72903" (UID: "19c8ef3a-6e3e-4178-ad3f-ccd093c72903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:34:05.554354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.554288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-data" (OuterVolumeSpecName: "istio-data") pod "19c8ef3a-6e3e-4178-ad3f-ccd093c72903" (UID: "19c8ef3a-6e3e-4178-ad3f-ccd093c72903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:34:05.554354 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.554329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19c8ef3a-6e3e-4178-ad3f-ccd093c72903" (UID: "19c8ef3a-6e3e-4178-ad3f-ccd093c72903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:34:05.554494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.554361    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:34:05.554585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.554499    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:34:05.554585 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.554513    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:34:05.555940 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.555911    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19c8ef3a-6e3e-4178-ad3f-ccd093c72903" (UID: "19c8ef3a-6e3e-4178-ad3f-ccd093c72903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:34:05.556062 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.556044    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-token" (OuterVolumeSpecName: "istio-token") pod "19c8ef3a-6e3e-4178-ad3f-ccd093c72903" (UID: "19c8ef3a-6e3e-4178-ad3f-ccd093c72903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:34:05.556564 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.556531    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-kube-api-access-z8d8g" (OuterVolumeSpecName: "kube-api-access-z8d8g") pod "19c8ef3a-6e3e-4178-ad3f-ccd093c72903" (UID: "19c8ef3a-6e3e-4178-ad3f-ccd093c72903"). InnerVolumeSpecName "kube-api-access-z8d8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:34:05.556622 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.556590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19c8ef3a-6e3e-4178-ad3f-ccd093c72903" (UID: "19c8ef3a-6e3e-4178-ad3f-ccd093c72903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:34:05.656004 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.655966    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:34:05.656004 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.655998    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8d8g\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-kube-api-access-z8d8g\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:34:05.656004 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.656011    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:34:05.656241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.656019    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:34:05.656241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.656027    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:34:05.656241 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.656034    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c8ef3a-6e3e-4178-ad3f-ccd093c72903-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:34:05.871071 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.871037    2580 generic.go:356] "Generic (PLEG): container finished" podID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerID="a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a" exitCode=0
Sep 10 23:34:05.871261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.871094    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"
Sep 10 23:34:05.871261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.871123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" event={"ID":"19c8ef3a-6e3e-4178-ad3f-ccd093c72903","Type":"ContainerDied","Data":"a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a"}
Sep 10 23:34:05.871261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.871185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf" event={"ID":"19c8ef3a-6e3e-4178-ad3f-ccd093c72903","Type":"ContainerDied","Data":"1e4e07ebd11350ea832c4bf95e35fac47e3ee5ac1bcc96ba9754a878a3db4fa6"}
Sep 10 23:34:05.871261 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.871216    2580 scope.go:117] "RemoveContainer" containerID="a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a"
Sep 10 23:34:05.881074 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.881058    2580 scope.go:117] "RemoveContainer" containerID="0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da"
Sep 10 23:34:05.889917 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.889898    2580 scope.go:117] "RemoveContainer" containerID="a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a"
Sep 10 23:34:05.890194 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:34:05.890175    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a\": container with ID starting with a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a not found: ID does not exist" containerID="a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a"
Sep 10 23:34:05.890242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.890207    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a"} err="failed to get container status \"a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a\": rpc error: code = NotFound desc = could not find container \"a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a\": container with ID starting with a32296918937b6955a5db875381427cb810851743cbfbddf0d2a2721fc4c909a not found: ID does not exist"
Sep 10 23:34:05.890242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.890230    2580 scope.go:117] "RemoveContainer" containerID="0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da"
Sep 10 23:34:05.890502 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:34:05.890481    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da\": container with ID starting with 0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da not found: ID does not exist" containerID="0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da"
Sep 10 23:34:05.890635 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.890511    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da"} err="failed to get container status \"0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da\": rpc error: code = NotFound desc = could not find container \"0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da\": container with ID starting with 0b9b8c82e1709247a41e6b1f4a81d4aaa787b3d40282eba037aff4abd7d904da not found: ID does not exist"
Sep 10 23:34:05.891792 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.891772    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"]
Sep 10 23:34:05.897264 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:05.897243    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-230464-openshift-default-684b5fffb6-tjwrf"]
Sep 10 23:34:06.834478 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:06.834448    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" path="/var/lib/kubelet/pods/19c8ef3a-6e3e-4178-ad3f-ccd093c72903/volumes"
Sep 10 23:34:17.076434 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:17.076400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:34:19.103962 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:19.103929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:19.159249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:19.159219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:19.216334 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:19.216297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:19.281120 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:19.281092    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:19.335490 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:19.335451    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:19.648237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:19.648206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:24.680450 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:24.680419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:28.866104 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:34:28.866063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:35:01.719858 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:01.719827    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:02.198698 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:02.198660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:07.936711 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:07.936677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:07.996217 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:07.996182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:08.170248 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:08.170214    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"]
Sep 10 23:35:08.170505 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:08.170480    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy" containerID="cri-o://a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56" gracePeriod=30
Sep 10 23:35:08.185588 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:08.185557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:08.781784 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:08.781749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:09.821994 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:09.821950    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 23:35:09.822358 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:09.822030    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:35:13.423421 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.423398    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:35:13.533943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.533834    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-socket\") pod \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") "
Sep 10 23:35:13.533943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.533895    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-data\") pod \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") "
Sep 10 23:35:13.534189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.533953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-podinfo\") pod \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") "
Sep 10 23:35:13.534189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.533979    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s458c\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-kube-api-access-s458c\") pod \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") "
Sep 10 23:35:13.534189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534035    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-credential-socket\") pod \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") "
Sep 10 23:35:13.534189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-certs\") pod \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") "
Sep 10 23:35:13.534189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istiod-ca-cert\") pod \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") "
Sep 10 23:35:13.534189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534167    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-token\") pod \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") "
Sep 10 23:35:13.534189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4144c225-7fe6-4e05-a1c9-9bc34b85e045" (UID: "4144c225-7fe6-4e05-a1c9-9bc34b85e045"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:13.534604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534198    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-envoy\") pod \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\" (UID: \"4144c225-7fe6-4e05-a1c9-9bc34b85e045\") "
Sep 10 23:35:13.534604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534224    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-data" (OuterVolumeSpecName: "istio-data") pod "4144c225-7fe6-4e05-a1c9-9bc34b85e045" (UID: "4144c225-7fe6-4e05-a1c9-9bc34b85e045"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:13.534604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4144c225-7fe6-4e05-a1c9-9bc34b85e045" (UID: "4144c225-7fe6-4e05-a1c9-9bc34b85e045"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:13.534604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4144c225-7fe6-4e05-a1c9-9bc34b85e045" (UID: "4144c225-7fe6-4e05-a1c9-9bc34b85e045"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:13.534604 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534531    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4144c225-7fe6-4e05-a1c9-9bc34b85e045" (UID: "4144c225-7fe6-4e05-a1c9-9bc34b85e045"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:13.534863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534674    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:13.534863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534693    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:13.534863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534707    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:13.534863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534721    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:13.534863 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.534732    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:13.536303 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.536279    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4144c225-7fe6-4e05-a1c9-9bc34b85e045" (UID: "4144c225-7fe6-4e05-a1c9-9bc34b85e045"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:35:13.536475 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.536456    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4144c225-7fe6-4e05-a1c9-9bc34b85e045" (UID: "4144c225-7fe6-4e05-a1c9-9bc34b85e045"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:13.537040 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.537017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-kube-api-access-s458c" (OuterVolumeSpecName: "kube-api-access-s458c") pod "4144c225-7fe6-4e05-a1c9-9bc34b85e045" (UID: "4144c225-7fe6-4e05-a1c9-9bc34b85e045"). InnerVolumeSpecName "kube-api-access-s458c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:13.537126 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.537045    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-token" (OuterVolumeSpecName: "istio-token") pod "4144c225-7fe6-4e05-a1c9-9bc34b85e045" (UID: "4144c225-7fe6-4e05-a1c9-9bc34b85e045"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:13.636117 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.636076    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:13.636117 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.636105    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s458c\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-kube-api-access-s458c\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:13.636117 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.636117    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:13.636117 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:13.636126    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4144c225-7fe6-4e05-a1c9-9bc34b85e045-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:14.267409 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.267378    2580 generic.go:356] "Generic (PLEG): container finished" podID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerID="a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56" exitCode=0
Sep 10 23:35:14.267664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.267432    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"
Sep 10 23:35:14.267664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.267441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" event={"ID":"4144c225-7fe6-4e05-a1c9-9bc34b85e045","Type":"ContainerDied","Data":"a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56"}
Sep 10 23:35:14.267664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.267486    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65" event={"ID":"4144c225-7fe6-4e05-a1c9-9bc34b85e045","Type":"ContainerDied","Data":"98a541b29e4ee98d2565dcd1e4395f18f608b424bd4a94aa7345aa6ac58aef89"}
Sep 10 23:35:14.267664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.267508    2580 scope.go:117] "RemoveContainer" containerID="a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56"
Sep 10 23:35:14.278217 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.278196    2580 scope.go:117] "RemoveContainer" containerID="5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065"
Sep 10 23:35:14.287286 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.287259    2580 scope.go:117] "RemoveContainer" containerID="a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56"
Sep 10 23:35:14.287599 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:35:14.287571    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56\": container with ID starting with a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56 not found: ID does not exist" containerID="a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56"
Sep 10 23:35:14.287661 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.287612    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56"} err="failed to get container status \"a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56\": rpc error: code = NotFound desc = could not find container \"a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56\": container with ID starting with a5a832ba119244028612df84770a7ccb6b7487b33d2219b0c29f0384f0886c56 not found: ID does not exist"
Sep 10 23:35:14.287661 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.287636    2580 scope.go:117] "RemoveContainer" containerID="5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065"
Sep 10 23:35:14.287929 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:35:14.287910    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065\": container with ID starting with 5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065 not found: ID does not exist" containerID="5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065"
Sep 10 23:35:14.287982 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.287963    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065"} err="failed to get container status \"5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065\": rpc error: code = NotFound desc = could not find container \"5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065\": container with ID starting with 5dca25c2bdfbbd545711caeeb67078ad8a2508ff24c39f0ad386168cb5846065 not found: ID does not exist"
Sep 10 23:35:14.288172 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.288157    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"]
Sep 10 23:35:14.291378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.291356    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ea2b31-openshift-default-86666c467d-zxv65"]
Sep 10 23:35:14.834344 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:14.834310    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" path="/var/lib/kubelet/pods/4144c225-7fe6-4e05-a1c9-9bc34b85e045/volumes"
Sep 10 23:35:18.989440 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:18.989397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:19.108862 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:19.108822    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:19.221266 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:19.221224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:19.289420 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:19.289334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:19.661002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:19.660968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:22.571505 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:22.571470    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:35:25.528106 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:25.528071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:25.629152 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:25.629117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:25.778165 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:25.778082    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"]
Sep 10 23:35:25.778392 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:25.778366    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy" containerID="cri-o://ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24" gracePeriod=30
Sep 10 23:35:25.794257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:25.794215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:26.378664 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:26.378629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:27.089289 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:27.089253    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 23:35:27.089672 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:27.089309    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:35:31.037495 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.037469    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:35:31.223611 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223572    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-data\") pod \"d67d0caa-3b8f-4700-b723-2eef33beb664\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") "
Sep 10 23:35:31.223788 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223627    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-token\") pod \"d67d0caa-3b8f-4700-b723-2eef33beb664\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") "
Sep 10 23:35:31.223788 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-socket\") pod \"d67d0caa-3b8f-4700-b723-2eef33beb664\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") "
Sep 10 23:35:31.223788 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-certs\") pod \"d67d0caa-3b8f-4700-b723-2eef33beb664\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") "
Sep 10 23:35:31.223943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-credential-socket\") pod \"d67d0caa-3b8f-4700-b723-2eef33beb664\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") "
Sep 10 23:35:31.223943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223819    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-envoy\") pod \"d67d0caa-3b8f-4700-b723-2eef33beb664\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") "
Sep 10 23:35:31.223943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223849    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt89r\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-kube-api-access-dt89r\") pod \"d67d0caa-3b8f-4700-b723-2eef33beb664\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") "
Sep 10 23:35:31.223943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223886    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-data" (OuterVolumeSpecName: "istio-data") pod "d67d0caa-3b8f-4700-b723-2eef33beb664" (UID: "d67d0caa-3b8f-4700-b723-2eef33beb664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:31.223943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223904    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-podinfo\") pod \"d67d0caa-3b8f-4700-b723-2eef33beb664\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") "
Sep 10 23:35:31.223943 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67d0caa-3b8f-4700-b723-2eef33beb664-istiod-ca-cert\") pod \"d67d0caa-3b8f-4700-b723-2eef33beb664\" (UID: \"d67d0caa-3b8f-4700-b723-2eef33beb664\") "
Sep 10 23:35:31.224246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.223984    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d67d0caa-3b8f-4700-b723-2eef33beb664" (UID: "d67d0caa-3b8f-4700-b723-2eef33beb664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:31.224246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.224066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d67d0caa-3b8f-4700-b723-2eef33beb664" (UID: "d67d0caa-3b8f-4700-b723-2eef33beb664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:31.224246 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.224151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d67d0caa-3b8f-4700-b723-2eef33beb664" (UID: "d67d0caa-3b8f-4700-b723-2eef33beb664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:31.224364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.224313    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:31.224364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.224332    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:31.224364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.224346    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:31.224364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.224360    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:31.224529 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.224342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67d0caa-3b8f-4700-b723-2eef33beb664-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d67d0caa-3b8f-4700-b723-2eef33beb664" (UID: "d67d0caa-3b8f-4700-b723-2eef33beb664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:31.226175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.226152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-token" (OuterVolumeSpecName: "istio-token") pod "d67d0caa-3b8f-4700-b723-2eef33beb664" (UID: "d67d0caa-3b8f-4700-b723-2eef33beb664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:31.226175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.226153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d67d0caa-3b8f-4700-b723-2eef33beb664" (UID: "d67d0caa-3b8f-4700-b723-2eef33beb664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:35:31.226374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.226176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-kube-api-access-dt89r" (OuterVolumeSpecName: "kube-api-access-dt89r") pod "d67d0caa-3b8f-4700-b723-2eef33beb664" (UID: "d67d0caa-3b8f-4700-b723-2eef33beb664"). InnerVolumeSpecName "kube-api-access-dt89r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:31.226424 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.226397    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d67d0caa-3b8f-4700-b723-2eef33beb664" (UID: "d67d0caa-3b8f-4700-b723-2eef33beb664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:31.325378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.325342    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:31.325378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.325372    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt89r\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-kube-api-access-dt89r\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:31.325378 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.325382    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:31.325648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.325390    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67d0caa-3b8f-4700-b723-2eef33beb664-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:31.325648 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.325399    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67d0caa-3b8f-4700-b723-2eef33beb664-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:31.366496 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.366459    2580 generic.go:356] "Generic (PLEG): container finished" podID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerID="ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24" exitCode=0
Sep 10 23:35:31.366676 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.366518    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"
Sep 10 23:35:31.366676 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.366569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" event={"ID":"d67d0caa-3b8f-4700-b723-2eef33beb664","Type":"ContainerDied","Data":"ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24"}
Sep 10 23:35:31.366676 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.366637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr" event={"ID":"d67d0caa-3b8f-4700-b723-2eef33beb664","Type":"ContainerDied","Data":"058e01de94305e431b802b1003e60ccd905c63b1f2bea97ca2fab2e01a04beb4"}
Sep 10 23:35:31.366676 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.366669    2580 scope.go:117] "RemoveContainer" containerID="ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24"
Sep 10 23:35:31.380703 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.380680    2580 scope.go:117] "RemoveContainer" containerID="13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe"
Sep 10 23:35:31.391647 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.391602    2580 scope.go:117] "RemoveContainer" containerID="ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24"
Sep 10 23:35:31.392007 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:35:31.391978    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24\": container with ID starting with ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24 not found: ID does not exist" containerID="ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24"
Sep 10 23:35:31.392231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.392054    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24"} err="failed to get container status \"ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24\": rpc error: code = NotFound desc = could not find container \"ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24\": container with ID starting with ab607c2e074090246f1aa24e9c18f1131704980fedde9e4f7e8824ff3e4e8e24 not found: ID does not exist"
Sep 10 23:35:31.392231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.392085    2580 scope.go:117] "RemoveContainer" containerID="13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe"
Sep 10 23:35:31.392231 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.392124    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"]
Sep 10 23:35:31.392645 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:35:31.392618    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe\": container with ID starting with 13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe not found: ID does not exist" containerID="13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe"
Sep 10 23:35:31.392741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.392652    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe"} err="failed to get container status \"13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe\": rpc error: code = NotFound desc = could not find container \"13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe\": container with ID starting with 13ca001ffca2561015d2b9480262d83392f0a674b46f119854589d2a4527a0fe not found: ID does not exist"
Sep 10 23:35:31.394616 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:31.394600    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-706b9d-openshift-default-674f6dcb6d-krhlr"]
Sep 10 23:35:32.834208 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:32.834179    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" path="/var/lib/kubelet/pods/d67d0caa-3b8f-4700-b723-2eef33beb664/volumes"
Sep 10 23:35:36.692663 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.692629    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"]
Sep 10 23:35:36.693126 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.692917    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m" podUID="15abecd3-d577-4522-90fb-9f40b8a12eba" containerName="maas-api" containerID="cri-o://42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64" gracePeriod=30
Sep 10 23:35:36.721823 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.721791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:36.797321 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.797265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:36.849783 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.849755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:36.918061 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.918026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:36.945307 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.945246    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:35:36.973047 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.973015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:36.974025 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.974000    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-maas-api-tls\") pod \"15abecd3-d577-4522-90fb-9f40b8a12eba\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") "
Sep 10 23:35:36.974179 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.974101    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abecd3-d577-4522-90fb-9f40b8a12eba-openshift-service-ca\") pod \"15abecd3-d577-4522-90fb-9f40b8a12eba\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") "
Sep 10 23:35:36.974179 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.974141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/15abecd3-d577-4522-90fb-9f40b8a12eba-kube-api-access-n472l\") pod \"15abecd3-d577-4522-90fb-9f40b8a12eba\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") "
Sep 10 23:35:36.974179 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.974160    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-metrics-certs\") pod \"15abecd3-d577-4522-90fb-9f40b8a12eba\" (UID: \"15abecd3-d577-4522-90fb-9f40b8a12eba\") "
Sep 10 23:35:36.974501 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.974478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15abecd3-d577-4522-90fb-9f40b8a12eba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15abecd3-d577-4522-90fb-9f40b8a12eba" (UID: "15abecd3-d577-4522-90fb-9f40b8a12eba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:36.976273 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.976243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15abecd3-d577-4522-90fb-9f40b8a12eba" (UID: "15abecd3-d577-4522-90fb-9f40b8a12eba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:35:36.976476 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.976457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15abecd3-d577-4522-90fb-9f40b8a12eba-kube-api-access-n472l" (OuterVolumeSpecName: "kube-api-access-n472l") pod "15abecd3-d577-4522-90fb-9f40b8a12eba" (UID: "15abecd3-d577-4522-90fb-9f40b8a12eba"). InnerVolumeSpecName "kube-api-access-n472l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:36.976530 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:36.976488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "15abecd3-d577-4522-90fb-9f40b8a12eba" (UID: "15abecd3-d577-4522-90fb-9f40b8a12eba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:35:37.075741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.075695    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abecd3-d577-4522-90fb-9f40b8a12eba-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:37.075954 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.075751    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/15abecd3-d577-4522-90fb-9f40b8a12eba-kube-api-access-n472l\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:37.075954 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.075767    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:37.075954 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.075781    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15abecd3-d577-4522-90fb-9f40b8a12eba-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:37.267941 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.267845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:37.404781 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.404742    2580 generic.go:356] "Generic (PLEG): container finished" podID="15abecd3-d577-4522-90fb-9f40b8a12eba" containerID="42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64" exitCode=0
Sep 10 23:35:37.404960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.404798    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"
Sep 10 23:35:37.404960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.404805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m" event={"ID":"15abecd3-d577-4522-90fb-9f40b8a12eba","Type":"ContainerDied","Data":"42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64"}
Sep 10 23:35:37.404960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.404857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m" event={"ID":"15abecd3-d577-4522-90fb-9f40b8a12eba","Type":"ContainerDied","Data":"9b0529f6e91e8e7bd2d143e06bacd5d3e296a108de6c16b451ba6b57fb5e6ad6"}
Sep 10 23:35:37.404960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.404888    2580 scope.go:117] "RemoveContainer" containerID="42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64"
Sep 10 23:35:37.414771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.414751    2580 scope.go:117] "RemoveContainer" containerID="42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64"
Sep 10 23:35:37.415026 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:35:37.415007    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64\": container with ID starting with 42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64 not found: ID does not exist" containerID="42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64"
Sep 10 23:35:37.415074 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.415034    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64"} err="failed to get container status \"42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64\": rpc error: code = NotFound desc = could not find container \"42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64\": container with ID starting with 42902311827aa8f97fe6223782184c824d054424ce789227802ac818bc892f64 not found: ID does not exist"
Sep 10 23:35:37.425276 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.425249    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"]
Sep 10 23:35:37.430797 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:37.430772    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c97f2-788bd4ccf9-2ds8m"]
Sep 10 23:35:38.834110 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:38.834078    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15abecd3-d577-4522-90fb-9f40b8a12eba" path="/var/lib/kubelet/pods/15abecd3-d577-4522-90fb-9f40b8a12eba/volumes"
Sep 10 23:35:39.581557 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:39.581507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:35:43.276793 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:43.276760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:43.376855 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:43.376818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:43.513641 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:43.513606    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"]
Sep 10 23:35:43.513885 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:43.513860    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy" containerID="cri-o://b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f" gracePeriod=30
Sep 10 23:35:43.529614 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:43.529520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:43.791595 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:43.791482    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 23:35:43.791740 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:43.791589    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:35:43.973145 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:43.973113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:48.773355 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.773331    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:35:48.889985 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.889956    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-socket\") pod \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") "
Sep 10 23:35:48.890146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.889991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istiod-ca-cert\") pod \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") "
Sep 10 23:35:48.890146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890035    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8bgx\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-kube-api-access-m8bgx\") pod \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") "
Sep 10 23:35:48.890146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-credential-socket\") pod \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") "
Sep 10 23:35:48.890146 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-certs\") pod \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") "
Sep 10 23:35:48.890305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-envoy\") pod \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") "
Sep 10 23:35:48.890305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5baf19b7-ab1d-46fd-8a29-91a219b45b11" (UID: "5baf19b7-ab1d-46fd-8a29-91a219b45b11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:48.890305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-data\") pod \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") "
Sep 10 23:35:48.890305 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-token\") pod \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") "
Sep 10 23:35:48.890453 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-podinfo\") pod \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\" (UID: \"5baf19b7-ab1d-46fd-8a29-91a219b45b11\") "
Sep 10 23:35:48.890511 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890458    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5baf19b7-ab1d-46fd-8a29-91a219b45b11" (UID: "5baf19b7-ab1d-46fd-8a29-91a219b45b11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:48.890511 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5baf19b7-ab1d-46fd-8a29-91a219b45b11" (UID: "5baf19b7-ab1d-46fd-8a29-91a219b45b11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:48.890771 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-data" (OuterVolumeSpecName: "istio-data") pod "5baf19b7-ab1d-46fd-8a29-91a219b45b11" (UID: "5baf19b7-ab1d-46fd-8a29-91a219b45b11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:48.890869 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.890776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5baf19b7-ab1d-46fd-8a29-91a219b45b11" (UID: "5baf19b7-ab1d-46fd-8a29-91a219b45b11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:48.891366 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.891344    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:48.891466 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.891371    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:48.891466 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.891384    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:48.891466 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.891398    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:48.891466 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.891412    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:48.892645 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.892614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-kube-api-access-m8bgx" (OuterVolumeSpecName: "kube-api-access-m8bgx") pod "5baf19b7-ab1d-46fd-8a29-91a219b45b11" (UID: "5baf19b7-ab1d-46fd-8a29-91a219b45b11"). InnerVolumeSpecName "kube-api-access-m8bgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:48.892645 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.892620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5baf19b7-ab1d-46fd-8a29-91a219b45b11" (UID: "5baf19b7-ab1d-46fd-8a29-91a219b45b11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:35:48.893135 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.893119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5baf19b7-ab1d-46fd-8a29-91a219b45b11" (UID: "5baf19b7-ab1d-46fd-8a29-91a219b45b11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:35:48.893284 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.893263    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-token" (OuterVolumeSpecName: "istio-token") pod "5baf19b7-ab1d-46fd-8a29-91a219b45b11" (UID: "5baf19b7-ab1d-46fd-8a29-91a219b45b11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:48.992497 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.992464    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8bgx\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-kube-api-access-m8bgx\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:48.992497 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.992491    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:48.992497 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.992500    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:48.992497 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:48.992509    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf19b7-ab1d-46fd-8a29-91a219b45b11-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:49.475475 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.475435    2580 generic.go:356] "Generic (PLEG): container finished" podID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerID="b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f" exitCode=0
Sep 10 23:35:49.475475 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.475479    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"
Sep 10 23:35:49.475761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.475505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" event={"ID":"5baf19b7-ab1d-46fd-8a29-91a219b45b11","Type":"ContainerDied","Data":"b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f"}
Sep 10 23:35:49.475761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.475581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88" event={"ID":"5baf19b7-ab1d-46fd-8a29-91a219b45b11","Type":"ContainerDied","Data":"e9ee3281461a53498ebd9b40e61b3acf1f67a0cd4a455808ec79b94415330321"}
Sep 10 23:35:49.475761 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.475614    2580 scope.go:117] "RemoveContainer" containerID="b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f"
Sep 10 23:35:49.485505 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.485486    2580 scope.go:117] "RemoveContainer" containerID="c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a"
Sep 10 23:35:49.493799 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.493776    2580 scope.go:117] "RemoveContainer" containerID="b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f"
Sep 10 23:35:49.494246 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:35:49.494189    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f\": container with ID starting with b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f not found: ID does not exist" containerID="b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f"
Sep 10 23:35:49.494352 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.494256    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f"} err="failed to get container status \"b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f\": rpc error: code = NotFound desc = could not find container \"b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f\": container with ID starting with b1e9b4b8c5bb07ad9e3b972df2b85a18f656f79137717d891a12018efb11c43f not found: ID does not exist"
Sep 10 23:35:49.494352 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.494281    2580 scope.go:117] "RemoveContainer" containerID="c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a"
Sep 10 23:35:49.494596 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:35:49.494568    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a\": container with ID starting with c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a not found: ID does not exist" containerID="c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a"
Sep 10 23:35:49.494690 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.494600    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a"} err="failed to get container status \"c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a\": rpc error: code = NotFound desc = could not find container \"c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a\": container with ID starting with c7dac0959433bd8b5c9db4b83322baabbfd05ebb3105ad21425b10b2f950710a not found: ID does not exist"
Sep 10 23:35:49.496662 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.496643    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"]
Sep 10 23:35:49.499750 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:49.499730    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c97f2-openshift-default-6669fbd6bb-9xp88"]
Sep 10 23:35:50.834237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:50.834206    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" path="/var/lib/kubelet/pods/5baf19b7-ab1d-46fd-8a29-91a219b45b11/volumes"
Sep 10 23:35:54.290087 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.290051    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"]
Sep 10 23:35:54.290459 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.290286    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl" podUID="28c0c3a5-72e2-4cbe-b293-7c59e125e001" containerName="maas-api" containerID="cri-o://ecfaa726f483763f4c7e726c6a2d57b657567f720035e510520625d36176675a" gracePeriod=30
Sep 10 23:35:54.313382 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.313353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:54.380091 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.380058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:54.449578 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.449520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:54.509216 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.509191    2580 generic.go:356] "Generic (PLEG): container finished" podID="28c0c3a5-72e2-4cbe-b293-7c59e125e001" containerID="ecfaa726f483763f4c7e726c6a2d57b657567f720035e510520625d36176675a" exitCode=0
Sep 10 23:35:54.509339 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.509265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl" event={"ID":"28c0c3a5-72e2-4cbe-b293-7c59e125e001","Type":"ContainerDied","Data":"ecfaa726f483763f4c7e726c6a2d57b657567f720035e510520625d36176675a"}
Sep 10 23:35:54.512071 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.512043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:54.528363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.528341    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:35:54.642631 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.642598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-metrics-certs\") pod \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") "
Sep 10 23:35:54.642825 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.642652    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t28rv\" (UniqueName: \"kubernetes.io/projected/28c0c3a5-72e2-4cbe-b293-7c59e125e001-kube-api-access-t28rv\") pod \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") "
Sep 10 23:35:54.642825 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.642689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c0c3a5-72e2-4cbe-b293-7c59e125e001-openshift-service-ca\") pod \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") "
Sep 10 23:35:54.642825 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.642718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-maas-api-tls\") pod \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\" (UID: \"28c0c3a5-72e2-4cbe-b293-7c59e125e001\") "
Sep 10 23:35:54.643161 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.643110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c0c3a5-72e2-4cbe-b293-7c59e125e001-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28c0c3a5-72e2-4cbe-b293-7c59e125e001" (UID: "28c0c3a5-72e2-4cbe-b293-7c59e125e001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:54.645012 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.644989    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c0c3a5-72e2-4cbe-b293-7c59e125e001-kube-api-access-t28rv" (OuterVolumeSpecName: "kube-api-access-t28rv") pod "28c0c3a5-72e2-4cbe-b293-7c59e125e001" (UID: "28c0c3a5-72e2-4cbe-b293-7c59e125e001"). InnerVolumeSpecName "kube-api-access-t28rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:54.645944 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.645921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28c0c3a5-72e2-4cbe-b293-7c59e125e001" (UID: "28c0c3a5-72e2-4cbe-b293-7c59e125e001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:35:54.645944 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.645937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "28c0c3a5-72e2-4cbe-b293-7c59e125e001" (UID: "28c0c3a5-72e2-4cbe-b293-7c59e125e001"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:35:54.744386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.744346    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t28rv\" (UniqueName: \"kubernetes.io/projected/28c0c3a5-72e2-4cbe-b293-7c59e125e001-kube-api-access-t28rv\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:54.744386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.744381    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c0c3a5-72e2-4cbe-b293-7c59e125e001-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:54.744386 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.744394    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:54.744660 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.744406    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c0c3a5-72e2-4cbe-b293-7c59e125e001-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:35:54.862445 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:54.862409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:55.516969 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:55.516943    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"
Sep 10 23:35:55.517364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:55.516984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl" event={"ID":"28c0c3a5-72e2-4cbe-b293-7c59e125e001","Type":"ContainerDied","Data":"6668bedb2eee0028d26a910d046d79dcf9952b97ed4af0f6fd5a9a59e0bb2505"}
Sep 10 23:35:55.517364 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:55.517059    2580 scope.go:117] "RemoveContainer" containerID="ecfaa726f483763f4c7e726c6a2d57b657567f720035e510520625d36176675a"
Sep 10 23:35:55.534148 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:55.534123    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"]
Sep 10 23:35:55.539955 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:55.539931    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-83be66-69c78f456f-mlqtl"]
Sep 10 23:35:56.834842 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:56.834809    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c0c3a5-72e2-4cbe-b293-7c59e125e001" path="/var/lib/kubelet/pods/28c0c3a5-72e2-4cbe-b293-7c59e125e001/volumes"
Sep 10 23:35:56.968983 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:35:56.968956    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:36:00.980019 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:00.979986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:01.080254 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:01.080223    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:01.268199 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:01.268119    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"]
Sep 10 23:36:01.268363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:01.268308    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy" containerID="cri-o://2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f" gracePeriod=30
Sep 10 23:36:01.270492 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:01.270462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:04.972571 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:04.972517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:36:06.203467 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.203423    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 23:36:06.203903 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.203505    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:36:06.523310 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.523285    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:36:06.581834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.581797    2580 generic.go:356] "Generic (PLEG): container finished" podID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerID="2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f" exitCode=0
Sep 10 23:36:06.582013 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.581848    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"
Sep 10 23:36:06.582013 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.581888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" event={"ID":"99dc4886-7b32-41a8-b83d-28ea1fa90740","Type":"ContainerDied","Data":"2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f"}
Sep 10 23:36:06.582013 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.581946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv" event={"ID":"99dc4886-7b32-41a8-b83d-28ea1fa90740","Type":"ContainerDied","Data":"ee802e1b985dd4006a3b820c60855ae97bc6e79b5b4bbd62b77abee1db1fc4d1"}
Sep 10 23:36:06.582013 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.581974    2580 scope.go:117] "RemoveContainer" containerID="2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f"
Sep 10 23:36:06.591061 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.591039    2580 scope.go:117] "RemoveContainer" containerID="d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d"
Sep 10 23:36:06.599687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.599669    2580 scope.go:117] "RemoveContainer" containerID="2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f"
Sep 10 23:36:06.599962 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:36:06.599943    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f\": container with ID starting with 2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f not found: ID does not exist" containerID="2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f"
Sep 10 23:36:06.600028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.599970    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f"} err="failed to get container status \"2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f\": rpc error: code = NotFound desc = could not find container \"2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f\": container with ID starting with 2a271fcc76da82d57931ae6f67aba44dbe96318d6770f35e076b5df9e503be3f not found: ID does not exist"
Sep 10 23:36:06.600028 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.599987    2580 scope.go:117] "RemoveContainer" containerID="d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d"
Sep 10 23:36:06.600248 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:36:06.600233    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d\": container with ID starting with d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d not found: ID does not exist" containerID="d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d"
Sep 10 23:36:06.600285 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.600250    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d"} err="failed to get container status \"d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d\": rpc error: code = NotFound desc = could not find container \"d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d\": container with ID starting with d30ae380b223973e4e8806c57823dc4bdb67e6401ae8554d86c2239e41fdf14d not found: ID does not exist"
Sep 10 23:36:06.652582 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652524    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh8g8\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-kube-api-access-fh8g8\") pod \"99dc4886-7b32-41a8-b83d-28ea1fa90740\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") "
Sep 10 23:36:06.652743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-envoy\") pod \"99dc4886-7b32-41a8-b83d-28ea1fa90740\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") "
Sep 10 23:36:06.652743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652618    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-credential-socket\") pod \"99dc4886-7b32-41a8-b83d-28ea1fa90740\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") "
Sep 10 23:36:06.652743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652669    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-podinfo\") pod \"99dc4886-7b32-41a8-b83d-28ea1fa90740\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") "
Sep 10 23:36:06.652743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-data\") pod \"99dc4886-7b32-41a8-b83d-28ea1fa90740\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") "
Sep 10 23:36:06.652743 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652733    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-socket\") pod \"99dc4886-7b32-41a8-b83d-28ea1fa90740\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") "
Sep 10 23:36:06.652979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99dc4886-7b32-41a8-b83d-28ea1fa90740-istiod-ca-cert\") pod \"99dc4886-7b32-41a8-b83d-28ea1fa90740\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") "
Sep 10 23:36:06.652979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-certs\") pod \"99dc4886-7b32-41a8-b83d-28ea1fa90740\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") "
Sep 10 23:36:06.652979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-token\") pod \"99dc4886-7b32-41a8-b83d-28ea1fa90740\" (UID: \"99dc4886-7b32-41a8-b83d-28ea1fa90740\") "
Sep 10 23:36:06.652979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99dc4886-7b32-41a8-b83d-28ea1fa90740" (UID: "99dc4886-7b32-41a8-b83d-28ea1fa90740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:36:06.653188 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.652984    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-data" (OuterVolumeSpecName: "istio-data") pod "99dc4886-7b32-41a8-b83d-28ea1fa90740" (UID: "99dc4886-7b32-41a8-b83d-28ea1fa90740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:36:06.653188 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.653099    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99dc4886-7b32-41a8-b83d-28ea1fa90740" (UID: "99dc4886-7b32-41a8-b83d-28ea1fa90740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:36:06.653188 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.653178    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-credential-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:36:06.653307 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.653196    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-data\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:36:06.653307 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.653224    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99dc4886-7b32-41a8-b83d-28ea1fa90740" (UID: "99dc4886-7b32-41a8-b83d-28ea1fa90740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:36:06.653307 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.653241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99dc4886-7b32-41a8-b83d-28ea1fa90740-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99dc4886-7b32-41a8-b83d-28ea1fa90740" (UID: "99dc4886-7b32-41a8-b83d-28ea1fa90740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:36:06.655050 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.655022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-token" (OuterVolumeSpecName: "istio-token") pod "99dc4886-7b32-41a8-b83d-28ea1fa90740" (UID: "99dc4886-7b32-41a8-b83d-28ea1fa90740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:36:06.655161 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.655057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99dc4886-7b32-41a8-b83d-28ea1fa90740" (UID: "99dc4886-7b32-41a8-b83d-28ea1fa90740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:36:06.655161 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.655093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-kube-api-access-fh8g8" (OuterVolumeSpecName: "kube-api-access-fh8g8") pod "99dc4886-7b32-41a8-b83d-28ea1fa90740" (UID: "99dc4886-7b32-41a8-b83d-28ea1fa90740"). InnerVolumeSpecName "kube-api-access-fh8g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:36:06.655342 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.655325    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99dc4886-7b32-41a8-b83d-28ea1fa90740" (UID: "99dc4886-7b32-41a8-b83d-28ea1fa90740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:36:06.754521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.754426    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh8g8\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-kube-api-access-fh8g8\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:36:06.754521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.754458    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-envoy\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:36:06.754521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.754467    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-podinfo\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:36:06.754521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.754475    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-socket\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:36:06.754521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.754483    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99dc4886-7b32-41a8-b83d-28ea1fa90740-istiod-ca-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:36:06.754521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.754491    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99dc4886-7b32-41a8-b83d-28ea1fa90740-workload-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:36:06.754521 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.754498    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99dc4886-7b32-41a8-b83d-28ea1fa90740-istio-token\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:36:06.898202 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.898173    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"]
Sep 10 23:36:06.902275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:06.902252    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-83be66-openshift-default-5bdd849c5d-fxzfv"]
Sep 10 23:36:08.833598 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:08.833562    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" path="/var/lib/kubelet/pods/99dc4886-7b32-41a8-b83d-28ea1fa90740/volumes"
Sep 10 23:36:34.387590 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:34.387518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:34.441445 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:36:34.441419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:05.042568 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042516    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-dvv28"]
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042872    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15abecd3-d577-4522-90fb-9f40b8a12eba" containerName="maas-api"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042882    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15abecd3-d577-4522-90fb-9f40b8a12eba" containerName="maas-api"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042890    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042897    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042903    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042909    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042916    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042921    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042927    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042931    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042945    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c0c3a5-72e2-4cbe-b293-7c59e125e001" containerName="maas-api"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042950    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c0c3a5-72e2-4cbe-b293-7c59e125e001" containerName="maas-api"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042955    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042961    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042967    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7be8ad8-6a4a-418f-8628-4b14814384fc" containerName="maas-api"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042972    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7be8ad8-6a4a-418f-8628-4b14814384fc" containerName="maas-api"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042978    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042983    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042989    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy"
Sep 10 23:37:05.042986 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.042994    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043006    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043011    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043024    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043076    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043083    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7be8ad8-6a4a-418f-8628-4b14814384fc" containerName="maas-api"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043088    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043094    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043101    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5baf19b7-ab1d-46fd-8a29-91a219b45b11" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043105    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043110    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043117    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043122    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c0c3a5-72e2-4cbe-b293-7c59e125e001" containerName="maas-api"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043127    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4144c225-7fe6-4e05-a1c9-9bc34b85e045" containerName="istio-proxy"
Sep 10 23:37:05.043609 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.043133    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15abecd3-d577-4522-90fb-9f40b8a12eba" containerName="maas-api"
Sep 10 23:37:05.045960 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.045944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.049006 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.048977    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 23:37:05.049006 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.048980    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fg9zd\""
Sep 10 23:37:05.049206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.049025    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 23:37:05.049206 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.048978    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 23:37:05.054949 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.054928    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-dvv28"]
Sep 10 23:37:05.087142 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.087094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7xwx\" (UniqueName: \"kubernetes.io/projected/5daf91fe-1d08-4c69-a781-1c22dce03665-kube-api-access-j7xwx\") pod \"maas-controller-7ff55dfbf9-dvv28\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") " pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.087142 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.087146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-webhook-cert\") pod \"maas-controller-7ff55dfbf9-dvv28\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") " pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.087361 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.087201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-metrics-certs\") pod \"maas-controller-7ff55dfbf9-dvv28\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") " pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.188503 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.188459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7xwx\" (UniqueName: \"kubernetes.io/projected/5daf91fe-1d08-4c69-a781-1c22dce03665-kube-api-access-j7xwx\") pod \"maas-controller-7ff55dfbf9-dvv28\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") " pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.188503 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.188512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-webhook-cert\") pod \"maas-controller-7ff55dfbf9-dvv28\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") " pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.188826 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.188665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-metrics-certs\") pod \"maas-controller-7ff55dfbf9-dvv28\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") " pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.191124 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.191101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-webhook-cert\") pod \"maas-controller-7ff55dfbf9-dvv28\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") " pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.191254 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.191232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-metrics-certs\") pod \"maas-controller-7ff55dfbf9-dvv28\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") " pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.197506 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.197477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7xwx\" (UniqueName: \"kubernetes.io/projected/5daf91fe-1d08-4c69-a781-1c22dce03665-kube-api-access-j7xwx\") pod \"maas-controller-7ff55dfbf9-dvv28\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") " pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.357628 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.357596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:05.632816 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.632732    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-dvv28"]
Sep 10 23:37:05.636216 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:37:05.636186    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5daf91fe_1d08_4c69_a781_1c22dce03665.slice/crio-7a488fc70c312ea2252282040d540d4a94d372efd4bb6bd8d0e81a501ed8d989 WatchSource:0}: Error finding container 7a488fc70c312ea2252282040d540d4a94d372efd4bb6bd8d0e81a501ed8d989: Status 404 returned error can't find the container with id 7a488fc70c312ea2252282040d540d4a94d372efd4bb6bd8d0e81a501ed8d989
Sep 10 23:37:05.905928 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:05.905838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28" event={"ID":"5daf91fe-1d08-4c69-a781-1c22dce03665","Type":"ContainerStarted","Data":"7a488fc70c312ea2252282040d540d4a94d372efd4bb6bd8d0e81a501ed8d989"}
Sep 10 23:37:07.919149 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:07.919118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28" event={"ID":"5daf91fe-1d08-4c69-a781-1c22dce03665","Type":"ContainerStarted","Data":"58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b"}
Sep 10 23:37:07.919509 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:07.919228    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:07.937705 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:07.937642    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28" podStartSLOduration=1.223033802 podStartE2EDuration="2.937622324s" totalImagesPullingTime="1.714588522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:07.935723338 +0000 UTC m=+2127.637356284" watchObservedRunningTime="2026-09-10 23:37:07.937622324 +0000 UTC m=+2127.639255267"
Sep 10 23:37:18.927167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:18.927131    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:28.323900 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:28.323866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:29.275237 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:29.275202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:37:30.237300 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.237265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:30.333220 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.333172    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-dvv28"]
Sep 10 23:37:30.333497 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.333468    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28" podUID="5daf91fe-1d08-4c69-a781-1c22dce03665" containerName="manager" containerID="cri-o://58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b" gracePeriod=30
Sep 10 23:37:30.581514 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.581487    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:30.718554 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.718521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7xwx\" (UniqueName: \"kubernetes.io/projected/5daf91fe-1d08-4c69-a781-1c22dce03665-kube-api-access-j7xwx\") pod \"5daf91fe-1d08-4c69-a781-1c22dce03665\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") "
Sep 10 23:37:30.718746 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.718611    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-webhook-cert\") pod \"5daf91fe-1d08-4c69-a781-1c22dce03665\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") "
Sep 10 23:37:30.718746 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.718649    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-metrics-certs\") pod \"5daf91fe-1d08-4c69-a781-1c22dce03665\" (UID: \"5daf91fe-1d08-4c69-a781-1c22dce03665\") "
Sep 10 23:37:30.720845 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.720811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5daf91fe-1d08-4c69-a781-1c22dce03665" (UID: "5daf91fe-1d08-4c69-a781-1c22dce03665"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:37:30.721377 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.721353    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5daf91fe-1d08-4c69-a781-1c22dce03665" (UID: "5daf91fe-1d08-4c69-a781-1c22dce03665"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:37:30.721470 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.721383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5daf91fe-1d08-4c69-a781-1c22dce03665-kube-api-access-j7xwx" (OuterVolumeSpecName: "kube-api-access-j7xwx") pod "5daf91fe-1d08-4c69-a781-1c22dce03665" (UID: "5daf91fe-1d08-4c69-a781-1c22dce03665"). InnerVolumeSpecName "kube-api-access-j7xwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:37:30.819505 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.819424    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7xwx\" (UniqueName: \"kubernetes.io/projected/5daf91fe-1d08-4c69-a781-1c22dce03665-kube-api-access-j7xwx\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:37:30.819505 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.819453    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-webhook-cert\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:37:30.819505 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:30.819466    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5daf91fe-1d08-4c69-a781-1c22dce03665-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:37:31.047957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.047920    2580 generic.go:356] "Generic (PLEG): container finished" podID="5daf91fe-1d08-4c69-a781-1c22dce03665" containerID="58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b" exitCode=0
Sep 10 23:37:31.048123 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.047972    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28"
Sep 10 23:37:31.048123 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.048014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28" event={"ID":"5daf91fe-1d08-4c69-a781-1c22dce03665","Type":"ContainerDied","Data":"58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b"}
Sep 10 23:37:31.048123 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.048065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-dvv28" event={"ID":"5daf91fe-1d08-4c69-a781-1c22dce03665","Type":"ContainerDied","Data":"7a488fc70c312ea2252282040d540d4a94d372efd4bb6bd8d0e81a501ed8d989"}
Sep 10 23:37:31.048123 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.048088    2580 scope.go:117] "RemoveContainer" containerID="58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b"
Sep 10 23:37:31.057474 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.057457    2580 scope.go:117] "RemoveContainer" containerID="58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b"
Sep 10 23:37:31.057775 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:37:31.057757    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b\": container with ID starting with 58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b not found: ID does not exist" containerID="58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b"
Sep 10 23:37:31.057833 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.057785    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b"} err="failed to get container status \"58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b\": rpc error: code = NotFound desc = could not find container \"58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b\": container with ID starting with 58cb1b51f79f3f6187edf4d2ce891cf79d94d0f8ac7280cfe4c076d180f0297b not found: ID does not exist"
Sep 10 23:37:31.064041 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.064017    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-dvv28"]
Sep 10 23:37:31.067458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.067433    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-dvv28"]
Sep 10 23:37:31.700835 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.700751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:31.811812 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.811781    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-v5l8j"]
Sep 10 23:37:31.812119 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.812108    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy"
Sep 10 23:37:31.812167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.812122    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c8ef3a-6e3e-4178-ad3f-ccd093c72903" containerName="istio-proxy"
Sep 10 23:37:31.812167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.812149    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daf91fe-1d08-4c69-a781-1c22dce03665" containerName="manager"
Sep 10 23:37:31.812167 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.812154    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daf91fe-1d08-4c69-a781-1c22dce03665" containerName="manager"
Sep 10 23:37:31.812264 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.812210    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67d0caa-3b8f-4700-b723-2eef33beb664" containerName="istio-proxy"
Sep 10 23:37:31.812264 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.812219    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5daf91fe-1d08-4c69-a781-1c22dce03665" containerName="manager"
Sep 10 23:37:31.812264 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.812226    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="99dc4886-7b32-41a8-b83d-28ea1fa90740" containerName="istio-proxy"
Sep 10 23:37:31.816312 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.816291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:31.818962 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.818935    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fg9zd\""
Sep 10 23:37:31.819076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.818935    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 23:37:31.819076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.818938    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 23:37:31.819076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.819042    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 23:37:31.824102 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.824084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-v5l8j"]
Sep 10 23:37:31.928422 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.928390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c910937-7ef8-4c55-a463-ff8df00889f4-webhook-cert\") pod \"maas-controller-7ff55dfbf9-v5l8j\" (UID: \"6c910937-7ef8-4c55-a463-ff8df00889f4\") " pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:31.928589 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.928503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xchhd\" (UniqueName: \"kubernetes.io/projected/6c910937-7ef8-4c55-a463-ff8df00889f4-kube-api-access-xchhd\") pod \"maas-controller-7ff55dfbf9-v5l8j\" (UID: \"6c910937-7ef8-4c55-a463-ff8df00889f4\") " pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:31.928659 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:31.928593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c910937-7ef8-4c55-a463-ff8df00889f4-metrics-certs\") pod \"maas-controller-7ff55dfbf9-v5l8j\" (UID: \"6c910937-7ef8-4c55-a463-ff8df00889f4\") " pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:32.029950 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:32.029859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xchhd\" (UniqueName: \"kubernetes.io/projected/6c910937-7ef8-4c55-a463-ff8df00889f4-kube-api-access-xchhd\") pod \"maas-controller-7ff55dfbf9-v5l8j\" (UID: \"6c910937-7ef8-4c55-a463-ff8df00889f4\") " pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:32.029950 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:32.029916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c910937-7ef8-4c55-a463-ff8df00889f4-metrics-certs\") pod \"maas-controller-7ff55dfbf9-v5l8j\" (UID: \"6c910937-7ef8-4c55-a463-ff8df00889f4\") " pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:32.030203 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:32.030031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c910937-7ef8-4c55-a463-ff8df00889f4-webhook-cert\") pod \"maas-controller-7ff55dfbf9-v5l8j\" (UID: \"6c910937-7ef8-4c55-a463-ff8df00889f4\") " pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:32.032458 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:32.032439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c910937-7ef8-4c55-a463-ff8df00889f4-metrics-certs\") pod \"maas-controller-7ff55dfbf9-v5l8j\" (UID: \"6c910937-7ef8-4c55-a463-ff8df00889f4\") " pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:32.032621 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:32.032601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c910937-7ef8-4c55-a463-ff8df00889f4-webhook-cert\") pod \"maas-controller-7ff55dfbf9-v5l8j\" (UID: \"6c910937-7ef8-4c55-a463-ff8df00889f4\") " pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:32.037551 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:32.037521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xchhd\" (UniqueName: \"kubernetes.io/projected/6c910937-7ef8-4c55-a463-ff8df00889f4-kube-api-access-xchhd\") pod \"maas-controller-7ff55dfbf9-v5l8j\" (UID: \"6c910937-7ef8-4c55-a463-ff8df00889f4\") " pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:32.127304 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:32.127268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:32.399372 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:32.399350    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-v5l8j"]
Sep 10 23:37:32.401401 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:37:32.401374    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c910937_7ef8_4c55_a463_ff8df00889f4.slice/crio-63a07115f8b77edb48305a759fd0b54305f65c41b3fc14740129026f59bdf94a WatchSource:0}: Error finding container 63a07115f8b77edb48305a759fd0b54305f65c41b3fc14740129026f59bdf94a: Status 404 returned error can't find the container with id 63a07115f8b77edb48305a759fd0b54305f65c41b3fc14740129026f59bdf94a
Sep 10 23:37:32.834469 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:32.834430    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5daf91fe-1d08-4c69-a781-1c22dce03665" path="/var/lib/kubelet/pods/5daf91fe-1d08-4c69-a781-1c22dce03665/volumes"
Sep 10 23:37:33.061658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:33.061567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j" event={"ID":"6c910937-7ef8-4c55-a463-ff8df00889f4","Type":"ContainerStarted","Data":"f2c6a14a9b3540fb312d81125583e2d31bb9d7aaa7f357d909fefa08b9eb1a3a"}
Sep 10 23:37:33.061658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:33.061637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j" event={"ID":"6c910937-7ef8-4c55-a463-ff8df00889f4","Type":"ContainerStarted","Data":"63a07115f8b77edb48305a759fd0b54305f65c41b3fc14740129026f59bdf94a"}
Sep 10 23:37:33.061844 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:33.061666    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:33.078654 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:33.078599    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j" podStartSLOduration=1.688508828 podStartE2EDuration="2.078583259s" totalImagesPullingTime="390.074431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:33.076405338 +0000 UTC m=+2152.778038283" watchObservedRunningTime="2026-09-10 23:37:33.078583259 +0000 UTC m=+2152.780216202"
Sep 10 23:37:44.068498 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:44.068467    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ff55dfbf9-v5l8j"
Sep 10 23:37:55.764454 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:55.764417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:37:58.169408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:58.169371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:59.325988 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:37:59.325955    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:03.567343 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:03.567305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:38:12.171766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:12.171725    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:38:29.174125 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:29.174096    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:38:35.068054 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:35.068018    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:38:36.512652 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:36.512618    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"]
Sep 10 23:38:36.513040 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:36.512854    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b" podUID="2a96ae29-51f6-411b-adc0-f9f737923d12" containerName="manager" containerID="cri-o://f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd" gracePeriod=10
Sep 10 23:38:37.567275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:37.567252    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:38:37.629979 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:37.629945    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgczn\" (UniqueName: \"kubernetes.io/projected/2a96ae29-51f6-411b-adc0-f9f737923d12-kube-api-access-pgczn\") pod \"2a96ae29-51f6-411b-adc0-f9f737923d12\" (UID: \"2a96ae29-51f6-411b-adc0-f9f737923d12\") "
Sep 10 23:38:37.630155 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:37.629991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a96ae29-51f6-411b-adc0-f9f737923d12-extensions-socket-volume\") pod \"2a96ae29-51f6-411b-adc0-f9f737923d12\" (UID: \"2a96ae29-51f6-411b-adc0-f9f737923d12\") "
Sep 10 23:38:37.630389 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:37.630356    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96ae29-51f6-411b-adc0-f9f737923d12-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2a96ae29-51f6-411b-adc0-f9f737923d12" (UID: "2a96ae29-51f6-411b-adc0-f9f737923d12"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:38:37.632210 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:37.632186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a96ae29-51f6-411b-adc0-f9f737923d12-kube-api-access-pgczn" (OuterVolumeSpecName: "kube-api-access-pgczn") pod "2a96ae29-51f6-411b-adc0-f9f737923d12" (UID: "2a96ae29-51f6-411b-adc0-f9f737923d12"). InnerVolumeSpecName "kube-api-access-pgczn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:37.731658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:37.731581    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgczn\" (UniqueName: \"kubernetes.io/projected/2a96ae29-51f6-411b-adc0-f9f737923d12-kube-api-access-pgczn\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:38:37.731658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:37.731607    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a96ae29-51f6-411b-adc0-f9f737923d12-extensions-socket-volume\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:38:38.418891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.418855    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a96ae29-51f6-411b-adc0-f9f737923d12" containerID="f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd" exitCode=0
Sep 10 23:38:38.419048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.418897    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"
Sep 10 23:38:38.419048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.418906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b" event={"ID":"2a96ae29-51f6-411b-adc0-f9f737923d12","Type":"ContainerDied","Data":"f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd"}
Sep 10 23:38:38.419048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.418948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b" event={"ID":"2a96ae29-51f6-411b-adc0-f9f737923d12","Type":"ContainerDied","Data":"9ef993ecd41aefb5f7a2cae575c982f8dbccc985d76b8f7d1a6c4a6b96cb49b3"}
Sep 10 23:38:38.419048 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.418971    2580 scope.go:117] "RemoveContainer" containerID="f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd"
Sep 10 23:38:38.431471 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.431448    2580 scope.go:117] "RemoveContainer" containerID="f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd"
Sep 10 23:38:38.431776 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:38:38.431753    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd\": container with ID starting with f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd not found: ID does not exist" containerID="f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd"
Sep 10 23:38:38.431842 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.431785    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd"} err="failed to get container status \"f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd\": rpc error: code = NotFound desc = could not find container \"f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd\": container with ID starting with f7399234531a6e4aa584f33b7801038f43db96debf32a1d21744761549415ecd not found: ID does not exist"
Sep 10 23:38:38.439387 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.439364    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"]
Sep 10 23:38:38.443464 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.443439    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dz55b"]
Sep 10 23:38:38.839591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:38.835449    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a96ae29-51f6-411b-adc0-f9f737923d12" path="/var/lib/kubelet/pods/2a96ae29-51f6-411b-adc0-f9f737923d12/volumes"
Sep 10 23:38:45.805681 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:45.805611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:45.857583 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:38:45.857534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:35.658122 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:35.658087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:37.200263 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.200224    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"]
Sep 10 23:39:37.200658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.200590    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a96ae29-51f6-411b-adc0-f9f737923d12" containerName="manager"
Sep 10 23:39:37.200658 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.200603    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a96ae29-51f6-411b-adc0-f9f737923d12" containerName="manager"
Sep 10 23:39:37.200732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.200679    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a96ae29-51f6-411b-adc0-f9f737923d12" containerName="manager"
Sep 10 23:39:37.203890 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.203874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:39:37.206084 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.206062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-thk95\""
Sep 10 23:39:37.223716 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.218680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"]
Sep 10 23:39:37.268623 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.268599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e6317af-f096-4f55-99f5-0770103327e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7lkpc\" (UID: \"9e6317af-f096-4f55-99f5-0770103327e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:39:37.268741 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.268665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc5pk\" (UniqueName: \"kubernetes.io/projected/9e6317af-f096-4f55-99f5-0770103327e0-kube-api-access-lc5pk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7lkpc\" (UID: \"9e6317af-f096-4f55-99f5-0770103327e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:39:37.369677 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.369647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e6317af-f096-4f55-99f5-0770103327e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7lkpc\" (UID: \"9e6317af-f096-4f55-99f5-0770103327e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:39:37.369830 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.369726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc5pk\" (UniqueName: \"kubernetes.io/projected/9e6317af-f096-4f55-99f5-0770103327e0-kube-api-access-lc5pk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7lkpc\" (UID: \"9e6317af-f096-4f55-99f5-0770103327e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:39:37.370033 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.370013    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e6317af-f096-4f55-99f5-0770103327e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7lkpc\" (UID: \"9e6317af-f096-4f55-99f5-0770103327e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:39:37.377464 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.377443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc5pk\" (UniqueName: \"kubernetes.io/projected/9e6317af-f096-4f55-99f5-0770103327e0-kube-api-access-lc5pk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7lkpc\" (UID: \"9e6317af-f096-4f55-99f5-0770103327e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:39:37.514332 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.514228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:39:37.788481 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:37.788453    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"]
Sep 10 23:39:37.791474 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:39:37.791449    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6317af_f096_4f55_99f5_0770103327e0.slice/crio-729ec7e4a898ea34a033722dde2b691233580e953aa0eec2b703005bec92535e WatchSource:0}: Error finding container 729ec7e4a898ea34a033722dde2b691233580e953aa0eec2b703005bec92535e: Status 404 returned error can't find the container with id 729ec7e4a898ea34a033722dde2b691233580e953aa0eec2b703005bec92535e
Sep 10 23:39:38.727846 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:38.727809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc" event={"ID":"9e6317af-f096-4f55-99f5-0770103327e0","Type":"ContainerStarted","Data":"a63897fa1def86fe81dc3f22cb004a314b36ce189676e54e336cddd29238c6bc"}
Sep 10 23:39:38.728294 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:38.727866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc" event={"ID":"9e6317af-f096-4f55-99f5-0770103327e0","Type":"ContainerStarted","Data":"729ec7e4a898ea34a033722dde2b691233580e953aa0eec2b703005bec92535e"}
Sep 10 23:39:38.728294 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:38.727947    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:39:38.751886 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:38.751834    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc" podStartSLOduration=1.751819259 podStartE2EDuration="1.751819259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:39:38.749531434 +0000 UTC m=+2278.451164387" watchObservedRunningTime="2026-09-10 23:39:38.751819259 +0000 UTC m=+2278.453452198"
Sep 10 23:39:49.736374 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:39:49.736345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7lkpc"
Sep 10 23:40:02.871926 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:40:02.871890    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:40:08.674410 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:40:08.674363    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:40:17.172828 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:40:17.172793    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:40:21.659559 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:40:21.659509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:24.579871 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:40:24.579835    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:40:25.277847 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:40:25.277814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:32.876105 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:40:32.876029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:40:46.256353 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:40:46.256319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:06.232007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:06.231969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:07.598498 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:07.598468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:09.477968 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:09.477931    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:41:16.975955 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:16.975919    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:41:21.397306 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.397272    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-zwwbm"]
Sep 10 23:41:21.400682 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.400662    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.403696 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.403666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5kfc5\""
Sep 10 23:41:21.403696 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.403679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 23:41:21.403696 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.403696    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:41:21.403932 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.403718    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:41:21.403932 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.403684    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:41:21.409667 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.409644    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"]
Sep 10 23:41:21.421494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.421460    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-zwwbm"]
Sep 10 23:41:21.421707 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.421610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.425390 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.425365    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"]
Sep 10 23:41:21.471834 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.471794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-maas-api-tls\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.472043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.471893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-openshift-service-ca\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.472043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.471928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51071269-610c-4ed7-b6ba-2bf2858a7f34-openshift-service-ca\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.472043 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.471972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhmn\" (UniqueName: \"kubernetes.io/projected/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-kube-api-access-wbhmn\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.472207 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.472068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-metrics-certs\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.472207 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.472121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8s29\" (UniqueName: \"kubernetes.io/projected/51071269-610c-4ed7-b6ba-2bf2858a7f34-kube-api-access-k8s29\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.472276 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.472207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-maas-api-tls\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.472322 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.472299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-metrics-certs\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.573770 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.573734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-metrics-certs\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.573978 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.573819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-maas-api-tls\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.573978 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.573855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-openshift-service-ca\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.573978 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.573883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51071269-610c-4ed7-b6ba-2bf2858a7f34-openshift-service-ca\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.573978 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.573916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhmn\" (UniqueName: \"kubernetes.io/projected/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-kube-api-access-wbhmn\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.573978 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.573949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-metrics-certs\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.574275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.573983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8s29\" (UniqueName: \"kubernetes.io/projected/51071269-610c-4ed7-b6ba-2bf2858a7f34-kube-api-access-k8s29\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.574275 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.574017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-maas-api-tls\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.574826 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.574793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-openshift-service-ca\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.575189 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.575135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51071269-610c-4ed7-b6ba-2bf2858a7f34-openshift-service-ca\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.576649 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.576626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-metrics-certs\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.577075 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.577049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-maas-api-tls\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.577175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.577086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-maas-api-tls\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.577175 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.577147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-metrics-certs\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.580980 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.580957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8s29\" (UniqueName: \"kubernetes.io/projected/51071269-610c-4ed7-b6ba-2bf2858a7f34-kube-api-access-k8s29\") pod \"maas-api-777777df97-zwwbm\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") " pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.581064 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.581049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhmn\" (UniqueName: \"kubernetes.io/projected/1fc01283-5cc7-423a-89a3-7e1e0d17dfe6-kube-api-access-wbhmn\") pod \"maas-api-9c7b54f65-mbwzp\" (UID: \"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.712795 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.712697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:21.733517 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.733490    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:21.991461 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.991429    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-zwwbm"]
Sep 10 23:41:21.992777 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:41:21.992752    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51071269_610c_4ed7_b6ba_2bf2858a7f34.slice/crio-7928ed8813f160b59e440bd597201d6702a0936e40c50f3694ff92c99aef9ee7 WatchSource:0}: Error finding container 7928ed8813f160b59e440bd597201d6702a0936e40c50f3694ff92c99aef9ee7: Status 404 returned error can't find the container with id 7928ed8813f160b59e440bd597201d6702a0936e40c50f3694ff92c99aef9ee7
Sep 10 23:41:21.994171 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:21.994151    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:41:22.022187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:22.022153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"]
Sep 10 23:41:22.025422 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:41:22.025383    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc01283_5cc7_423a_89a3_7e1e0d17dfe6.slice/crio-f2500865e36a675236ef03466e416aa853172d7b5bd22fc5ba2552a274b7ada6 WatchSource:0}: Error finding container f2500865e36a675236ef03466e416aa853172d7b5bd22fc5ba2552a274b7ada6: Status 404 returned error can't find the container with id f2500865e36a675236ef03466e416aa853172d7b5bd22fc5ba2552a274b7ada6
Sep 10 23:41:22.271156 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:22.271070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:41:22.290089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:22.290045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm" event={"ID":"51071269-610c-4ed7-b6ba-2bf2858a7f34","Type":"ContainerStarted","Data":"7928ed8813f160b59e440bd597201d6702a0936e40c50f3694ff92c99aef9ee7"}
Sep 10 23:41:22.291286 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:22.291250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp" event={"ID":"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6","Type":"ContainerStarted","Data":"f2500865e36a675236ef03466e416aa853172d7b5bd22fc5ba2552a274b7ada6"}
Sep 10 23:41:23.298565 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:23.298504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm" event={"ID":"51071269-610c-4ed7-b6ba-2bf2858a7f34","Type":"ContainerStarted","Data":"faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139"}
Sep 10 23:41:23.299015 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:23.298621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:23.300117 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:23.300092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp" event={"ID":"1fc01283-5cc7-423a-89a3-7e1e0d17dfe6","Type":"ContainerStarted","Data":"409634e85fa93ebc745be182aaca6f39d47ce49c855a7d292eb73d6a77a8d9bc"}
Sep 10 23:41:23.300225 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:23.300136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:23.317324 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:23.317274    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm" podStartSLOduration=1.8169873189999999 podStartE2EDuration="2.317257439s" totalImagesPullingTime="500.27012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:23.314624403 +0000 UTC m=+2383.016257348" watchObservedRunningTime="2026-09-10 23:41:23.317257439 +0000 UTC m=+2383.018890381"
Sep 10 23:41:23.331925 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:23.331867    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp" podStartSLOduration=1.877162124 podStartE2EDuration="2.331849456s" totalImagesPullingTime="454.687332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:23.329382837 +0000 UTC m=+2383.031015772" watchObservedRunningTime="2026-09-10 23:41:23.331849456 +0000 UTC m=+2383.033482398"
Sep 10 23:41:29.274277 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.274242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:41:29.310858 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.310833    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-mbwzp"
Sep 10 23:41:29.311257 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.311234    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:29.378384 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.378351    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-zwwbm"]
Sep 10 23:41:29.378624 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.378597    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm" podUID="51071269-610c-4ed7-b6ba-2bf2858a7f34" containerName="maas-api" containerID="cri-o://faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139" gracePeriod=30
Sep 10 23:41:29.620269 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.620246    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:29.749463 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.749431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-metrics-certs\") pod \"51071269-610c-4ed7-b6ba-2bf2858a7f34\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") "
Sep 10 23:41:29.749666 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.749491    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8s29\" (UniqueName: \"kubernetes.io/projected/51071269-610c-4ed7-b6ba-2bf2858a7f34-kube-api-access-k8s29\") pod \"51071269-610c-4ed7-b6ba-2bf2858a7f34\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") "
Sep 10 23:41:29.749666 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.749522    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-maas-api-tls\") pod \"51071269-610c-4ed7-b6ba-2bf2858a7f34\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") "
Sep 10 23:41:29.749666 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.749564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51071269-610c-4ed7-b6ba-2bf2858a7f34-openshift-service-ca\") pod \"51071269-610c-4ed7-b6ba-2bf2858a7f34\" (UID: \"51071269-610c-4ed7-b6ba-2bf2858a7f34\") "
Sep 10 23:41:29.750096 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.750069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51071269-610c-4ed7-b6ba-2bf2858a7f34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51071269-610c-4ed7-b6ba-2bf2858a7f34" (UID: "51071269-610c-4ed7-b6ba-2bf2858a7f34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:41:29.751887 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.751862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51071269-610c-4ed7-b6ba-2bf2858a7f34" (UID: "51071269-610c-4ed7-b6ba-2bf2858a7f34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:41:29.752182 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.752153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "51071269-610c-4ed7-b6ba-2bf2858a7f34" (UID: "51071269-610c-4ed7-b6ba-2bf2858a7f34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:41:29.752298 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.752179    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51071269-610c-4ed7-b6ba-2bf2858a7f34-kube-api-access-k8s29" (OuterVolumeSpecName: "kube-api-access-k8s29") pod "51071269-610c-4ed7-b6ba-2bf2858a7f34" (UID: "51071269-610c-4ed7-b6ba-2bf2858a7f34"). InnerVolumeSpecName "kube-api-access-k8s29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:41:29.850911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.850876    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-maas-api-tls\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:41:29.850911 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.850909    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51071269-610c-4ed7-b6ba-2bf2858a7f34-openshift-service-ca\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:41:29.851112 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.850924    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51071269-610c-4ed7-b6ba-2bf2858a7f34-metrics-certs\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:41:29.851112 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:29.850936    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8s29\" (UniqueName: \"kubernetes.io/projected/51071269-610c-4ed7-b6ba-2bf2858a7f34-kube-api-access-k8s29\") on node \"ip-10-0-135-96.ec2.internal\" DevicePath \"\""
Sep 10 23:41:30.346362 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.346331    2580 generic.go:356] "Generic (PLEG): container finished" podID="51071269-610c-4ed7-b6ba-2bf2858a7f34" containerID="faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139" exitCode=0
Sep 10 23:41:30.346902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.346381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm" event={"ID":"51071269-610c-4ed7-b6ba-2bf2858a7f34","Type":"ContainerDied","Data":"faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139"}
Sep 10 23:41:30.346902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.346401    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm"
Sep 10 23:41:30.346902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.346438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777777df97-zwwbm" event={"ID":"51071269-610c-4ed7-b6ba-2bf2858a7f34","Type":"ContainerDied","Data":"7928ed8813f160b59e440bd597201d6702a0936e40c50f3694ff92c99aef9ee7"}
Sep 10 23:41:30.346902 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.346460    2580 scope.go:117] "RemoveContainer" containerID="faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139"
Sep 10 23:41:30.358195 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.358169    2580 scope.go:117] "RemoveContainer" containerID="faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139"
Sep 10 23:41:30.358615 ip-10-0-135-96 kubenswrapper[2580]: E0910 23:41:30.358528    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139\": container with ID starting with faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139 not found: ID does not exist" containerID="faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139"
Sep 10 23:41:30.358615 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.358583    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139"} err="failed to get container status \"faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139\": rpc error: code = NotFound desc = could not find container \"faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139\": container with ID starting with faaa70600a91368713cbcec4e9dc14f987f0da5e563a654ca70753e111e6c139 not found: ID does not exist"
Sep 10 23:41:30.370862 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.370829    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-zwwbm"]
Sep 10 23:41:30.376085 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.376058    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-zwwbm"]
Sep 10 23:41:30.835733 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:30.835642    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51071269-610c-4ed7-b6ba-2bf2858a7f34" path="/var/lib/kubelet/pods/51071269-610c-4ed7-b6ba-2bf2858a7f34/volumes"
Sep 10 23:41:35.171242 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:35.171204    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:41:44.478970 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:44.478936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:41:49.070089 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:49.070049    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9kslx"]
Sep 10 23:41:57.309709 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:57.309676    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4d964_5158dc1c-1f72-4bf4-8fe4-7004ceaf4060/manager/0.log"
Sep 10 23:41:57.552203 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:57.552167    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ff55dfbf9-v5l8j_6c910937-7ef8-4c55-a463-ff8df00889f4/manager/0.log"
Sep 10 23:41:57.881935 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:57.881907    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-227km_124dfcd6-e648-4a84-8297-27ca3faf1fde/copy-manifests/0.log"
Sep 10 23:41:57.894325 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:57.894296    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-227km_124dfcd6-e648-4a84-8297-27ca3faf1fde/manager/0.log"
Sep 10 23:41:58.569213 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:41:58.569183    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-9c7b54f65-mbwzp_1fc01283-5cc7-423a-89a3-7e1e0d17dfe6/maas-api/0.log"
Sep 10 23:42:00.069675 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:00.069647    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-7lkpc_9e6317af-f096-4f55-99f5-0770103327e0/manager/0.log"
Sep 10 23:42:00.192448 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:00.192419    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-9kslx_17d0b358-cf06-4070-a8e1-1c2b83e9edba/limitador/0.log"
Sep 10 23:42:00.638754 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:00.638724    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w_56044d1b-0c10-40ea-8188-e999b0f669b8/istio-proxy/0.log"
Sep 10 23:42:01.092554 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:01.092459    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-j687t_418cef94-9a26-4377-b280-7edba1711bdb/istio-proxy/1.log"
Sep 10 23:42:02.405198 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:02.405114    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-brkkz_39d38071-e474-463b-8e46-8761f3796170/main/0.log"
Sep 10 23:42:07.981376 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:07.981342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:08.040751 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:08.040718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:10.259269 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:10.259226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:10.315889 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:10.315863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:10.933732 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:10.933703    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zsct5_1071ba46-0db7-4471-8c45-33cd7ca6dc8e/global-pull-secret-syncer/0.log"
Sep 10 23:42:10.979408 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:10.979375    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2hghp_4ded8dd6-e929-46c4-a995-649649748c61/konnectivity-agent/0.log"
Sep 10 23:42:11.102443 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:11.102416    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-96.ec2.internal_6c72a50c000c2c43d429e4d0ccd75117/haproxy/0.log"
Sep 10 23:42:16.221193 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:16.221152    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-7lkpc_9e6317af-f096-4f55-99f5-0770103327e0/manager/0.log"
Sep 10 23:42:16.250166 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:16.250140    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-9kslx_17d0b358-cf06-4070-a8e1-1c2b83e9edba/limitador/0.log"
Sep 10 23:42:18.347147 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:18.347116    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cf87649cf-7cjdm_151ea0af-d668-4ea1-aefc-962ac0de568c/metrics-server/0.log"
Sep 10 23:42:18.585676 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:18.585638    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zgfl2_f8e3b7ef-9920-4542-84ce-be45a7805f17/node-exporter/0.log"
Sep 10 23:42:18.610268 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:18.610196    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zgfl2_f8e3b7ef-9920-4542-84ce-be45a7805f17/kube-rbac-proxy/0.log"
Sep 10 23:42:18.635030 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:18.635000    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zgfl2_f8e3b7ef-9920-4542-84ce-be45a7805f17/init-textfile/0.log"
Sep 10 23:42:18.931334 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:18.931256    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-8f82g_9cd9abb2-9e7b-4a6f-81a2-695305347b2e/prometheus-operator/0.log"
Sep 10 23:42:18.952846 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:18.952817    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-8f82g_9cd9abb2-9e7b-4a6f-81a2-695305347b2e/kube-rbac-proxy/0.log"
Sep 10 23:42:18.975645 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:18.975622    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-ntjzf_4fae47ba-f91a-48dd-8986-2649ff5f023c/prometheus-operator-admission-webhook/0.log"
Sep 10 23:42:19.096406 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:19.096380    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c5dd6dbd-czd86_3bc0b864-2ec0-4354-9937-fba124f98cf0/thanos-query/0.log"
Sep 10 23:42:19.122529 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:19.122495    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c5dd6dbd-czd86_3bc0b864-2ec0-4354-9937-fba124f98cf0/kube-rbac-proxy-web/0.log"
Sep 10 23:42:19.142748 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:19.142723    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c5dd6dbd-czd86_3bc0b864-2ec0-4354-9937-fba124f98cf0/kube-rbac-proxy/0.log"
Sep 10 23:42:19.176165 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:19.176143    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c5dd6dbd-czd86_3bc0b864-2ec0-4354-9937-fba124f98cf0/prom-label-proxy/0.log"
Sep 10 23:42:19.195907 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:19.195850    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c5dd6dbd-czd86_3bc0b864-2ec0-4354-9937-fba124f98cf0/kube-rbac-proxy-rules/0.log"
Sep 10 23:42:19.215725 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:19.215702    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c5dd6dbd-czd86_3bc0b864-2ec0-4354-9937-fba124f98cf0/kube-rbac-proxy-metrics/0.log"
Sep 10 23:42:20.605668 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:20.605639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-p6sdw_38ad9cd3-c937-4c4a-bd49-ee4665ce8627/networking-console-plugin/0.log"
Sep 10 23:42:21.775326 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.775294    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"]
Sep 10 23:42:21.775957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.775837    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51071269-610c-4ed7-b6ba-2bf2858a7f34" containerName="maas-api"
Sep 10 23:42:21.775957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.775855    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51071269-610c-4ed7-b6ba-2bf2858a7f34" containerName="maas-api"
Sep 10 23:42:21.775957 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.775953    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51071269-610c-4ed7-b6ba-2bf2858a7f34" containerName="maas-api"
Sep 10 23:42:21.779112 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.779090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:21.781363 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.781346    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xqw8\"/\"kube-root-ca.crt\""
Sep 10 23:42:21.782142 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.782124    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5xqw8\"/\"default-dockercfg-8m8wk\""
Sep 10 23:42:21.782244 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.782205    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xqw8\"/\"openshift-service-ca.crt\""
Sep 10 23:42:21.788973 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.788952    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"]
Sep 10 23:42:21.802465 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.802441    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-df766899b-4pdnr_1f19623b-f7b4-4b23-a3e0-39c5adff27f7/console/0.log"
Sep 10 23:42:21.831749 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.831721    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-t68b8_6725a188-ca0f-4af0-832f-172a87277628/download-server/0.log"
Sep 10 23:42:21.924510 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.924483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-lib-modules\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:21.924687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.924525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-proc\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:21.924687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.924578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-podres\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:21.924687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.924623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-sys\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:21.924687 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:21.924643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdtsf\" (UniqueName: \"kubernetes.io/projected/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-kube-api-access-rdtsf\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.025210 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.025173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-lib-modules\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.025393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.025221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-proc\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.025393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.025244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-podres\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.025393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.025275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-sys\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.025393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.025296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdtsf\" (UniqueName: \"kubernetes.io/projected/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-kube-api-access-rdtsf\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.025393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.025304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-proc\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.025393 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.025358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-lib-modules\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.025635 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.025404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-sys\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.025635 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.025422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-podres\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.033076 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.033056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdtsf\" (UniqueName: \"kubernetes.io/projected/d4717900-368f-43c6-9e0f-e4b9cb3d84e9-kube-api-access-rdtsf\") pod \"perf-node-gather-daemonset-gf9c6\" (UID: \"d4717900-368f-43c6-9e0f-e4b9cb3d84e9\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.089598 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.089572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.359804 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.359776    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"]
Sep 10 23:42:22.361750 ip-10-0-135-96 kubenswrapper[2580]: W0910 23:42:22.361722    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4717900_368f_43c6_9e0f_e4b9cb3d84e9.slice/crio-cc764f2648684bf626342aecab4903208c2a8ed2edadc0cbaa1fa57503bb38c1 WatchSource:0}: Error finding container cc764f2648684bf626342aecab4903208c2a8ed2edadc0cbaa1fa57503bb38c1: Status 404 returned error can't find the container with id cc764f2648684bf626342aecab4903208c2a8ed2edadc0cbaa1fa57503bb38c1
Sep 10 23:42:22.639038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.638951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6" event={"ID":"d4717900-368f-43c6-9e0f-e4b9cb3d84e9","Type":"ContainerStarted","Data":"5e6d08f0ba02c681c3a3a5c58e47fad31d00ca349988383238f1da2ab07c04fc"}
Sep 10 23:42:22.639038 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.639004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6" event={"ID":"d4717900-368f-43c6-9e0f-e4b9cb3d84e9","Type":"ContainerStarted","Data":"cc764f2648684bf626342aecab4903208c2a8ed2edadc0cbaa1fa57503bb38c1"}
Sep 10 23:42:22.639221 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.639088    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:22.653307 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:22.653250    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6" podStartSLOduration=1.65323223 podStartE2EDuration="1.65323223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:22.65290546 +0000 UTC m=+2442.354538403" watchObservedRunningTime="2026-09-10 23:42:22.65323223 +0000 UTC m=+2442.354865173"
Sep 10 23:42:23.680959 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:23.680916    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hstjw_6204e483-a2f1-4263-ae88-066b9e5d34ea/dns/0.log"
Sep 10 23:42:23.701766 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:23.701727    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hstjw_6204e483-a2f1-4263-ae88-066b9e5d34ea/kube-rbac-proxy/0.log"
Sep 10 23:42:23.788070 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:23.788043    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dh6gp_19129d1e-38ab-4ac7-b5ee-06d696663198/dns-node-resolver/0.log"
Sep 10 23:42:24.335891 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:24.335853    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77d46bf7c-dtlxq_8f80e44d-eb36-4008-9f71-89627dc96a5e/registry/0.log"
Sep 10 23:42:24.374983 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:24.374957    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hs27n_575fae60-bced-4137-b2c8-9555e0e59a1b/node-ca/0.log"
Sep 10 23:42:25.308494 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:25.308460    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w_56044d1b-0c10-40ea-8188-e999b0f669b8/istio-proxy/0.log"
Sep 10 23:42:25.664132 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:25.664100    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-j687t_418cef94-9a26-4377-b280-7edba1711bdb/istio-proxy/0.log"
Sep 10 23:42:25.666746 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:25.666725    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-j687t_418cef94-9a26-4377-b280-7edba1711bdb/istio-proxy/1.log"
Sep 10 23:42:26.703876 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:26.703850    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vmqvh_106e6c90-7df4-4ff0-954d-81ec5c2b397e/serve-healthcheck-canary/0.log"
Sep 10 23:42:27.311711 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:27.311672    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvvkj_399acb8d-754f-4775-a5f8-508412508334/kube-rbac-proxy/0.log"
Sep 10 23:42:27.329904 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:27.329874    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvvkj_399acb8d-754f-4775-a5f8-508412508334/exporter/0.log"
Sep 10 23:42:27.351084 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:27.351062    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvvkj_399acb8d-754f-4775-a5f8-508412508334/extractor/0.log"
Sep 10 23:42:28.654594 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:28.654568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-gf9c6"
Sep 10 23:42:29.413779 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:29.413748    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4d964_5158dc1c-1f72-4bf4-8fe4-7004ceaf4060/manager/0.log"
Sep 10 23:42:29.693225 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:29.693143    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ff55dfbf9-v5l8j_6c910937-7ef8-4c55-a463-ff8df00889f4/manager/0.log"
Sep 10 23:42:29.860739 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:29.860712    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-227km_124dfcd6-e648-4a84-8297-27ca3faf1fde/manager/0.log"
Sep 10 23:42:29.875069 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:29.875049    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-227km_124dfcd6-e648-4a84-8297-27ca3faf1fde/copy-manifests/0.log"
Sep 10 23:42:31.392883 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:31.392854    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-559dcbdd76-zb5wc_7028e57c-8023-436c-b6cb-56388f386a9c/manager/0.log"
Sep 10 23:42:31.415729 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:31.415698    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fcgph_b48108d1-fd79-45c1-867f-abdf51f5b24e/openshift-lws-operator/0.log"
Sep 10 23:42:32.347002 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:32.346973    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w_56044d1b-0c10-40ea-8188-e999b0f669b8/istio-proxy/0.log"
Sep 10 23:42:32.660634 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:32.660524    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-j687t_418cef94-9a26-4377-b280-7edba1711bdb/istio-proxy/0.log"
Sep 10 23:42:32.663007 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:32.662989    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-j687t_418cef94-9a26-4377-b280-7edba1711bdb/istio-proxy/1.log"
Sep 10 23:42:34.335371 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:34.335343    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cb7v7w_56044d1b-0c10-40ea-8188-e999b0f669b8/istio-proxy/0.log"
Sep 10 23:42:34.667649 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:34.667612    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-j687t_418cef94-9a26-4377-b280-7edba1711bdb/istio-proxy/0.log"
Sep 10 23:42:34.669799 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:34.669779    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-j687t_418cef94-9a26-4377-b280-7edba1711bdb/istio-proxy/1.log"
Sep 10 23:42:36.582796 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:36.582762    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-4bkhs_54c128a2-2ed8-40e7-bd25-010f892e9f63/migrator/0.log"
Sep 10 23:42:36.616733 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:36.616697    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-4bkhs_54c128a2-2ed8-40e7-bd25-010f892e9f63/graceful-termination/0.log"
Sep 10 23:42:37.939093 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:37.939066    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crvrl_ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69/kube-multus-additional-cni-plugins/0.log"
Sep 10 23:42:37.959591 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:37.959536    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crvrl_ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69/egress-router-binary-copy/0.log"
Sep 10 23:42:37.978187 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:37.978139    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crvrl_ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69/cni-plugins/0.log"
Sep 10 23:42:37.996396 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:37.996367    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crvrl_ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69/bond-cni-plugin/0.log"
Sep 10 23:42:38.022249 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:38.022204    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crvrl_ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69/routeoverride-cni/0.log"
Sep 10 23:42:38.042695 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:38.042670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crvrl_ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69/whereabouts-cni-bincopy/0.log"
Sep 10 23:42:38.062600 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:38.062577    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crvrl_ff5a40bf-f0d6-45d7-9d2e-c0a7f1169c69/whereabouts-cni/0.log"
Sep 10 23:42:38.442252 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:38.442213    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vqtcn_56ea190a-f9a3-48ff-89b4-6899da36d6da/kube-multus/0.log"
Sep 10 23:42:38.503101 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:38.503075    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v26f5_6b7b8f65-5f39-4ce7-b892-c70e379f3437/network-metrics-daemon/0.log"
Sep 10 23:42:38.518010 ip-10-0-135-96 kubenswrapper[2580]: I0910 23:42:38.517989    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v26f5_6b7b8f65-5f39-4ce7-b892-c70e379f3437/kube-rbac-proxy/0.log"