Aug 04 00:00:31.697293 ip-10-0-132-168 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 00:00:31.697305 ip-10-0-132-168 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 00:00:31.697314 ip-10-0-132-168 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 00:00:31.697626 ip-10-0-132-168 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 00:00:41.704635 ip-10-0-132-168 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 04 00:00:41.704655 ip-10-0-132-168 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9a24ddcda29e4adda878259c1ba8754b -- Aug 04 00:02:41.200510 ip-10-0-132-168 systemd[1]: Starting Kubernetes Kubelet... Aug 04 00:02:41.655559 ip-10-0-132-168 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 00:02:41.655559 ip-10-0-132-168 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 00:02:41.655559 ip-10-0-132-168 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 00:02:41.655559 ip-10-0-132-168 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 00:02:41.655559 ip-10-0-132-168 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 00:02:41.657852 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.657733 2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 00:02:41.662683 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662667 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:41.662683 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662683 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662687 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662692 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662698 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662702 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662706 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662710 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662714 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662717 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662720 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662724 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662727 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662730 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662733 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662736 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662739 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662743 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662745 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662748 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:41.662762 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662751 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662755 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662758 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662761 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662764 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662767 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662770 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662773 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662776 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662779 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662782 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662785 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662788 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662791 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662794 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662798 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662803 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662807 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662810 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662813 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:41.663355 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662816 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662818 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662821 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662824 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662827 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662829 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662832 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662835 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662838 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662840 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662843 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662846 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662849 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662852 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662856 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662860 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662863 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662866 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662869 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662872 2552 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:41.663873 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662875 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662877 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662880 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662884 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662888 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662891 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662894 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662897 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662901 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662904 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662907 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662910 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662913 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662916 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662919 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662922 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662925 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662928 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662945 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662948 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:41.664433 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662951 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662954 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662957 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662960 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662962 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662965 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662969 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662973 2552 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662977 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662980 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662983 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662986 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662989 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662992 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662994 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.662997 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663000 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663003 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663006 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:41.664988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663763 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663770 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663773 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663776 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663779 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663782 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663785 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663789 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663792 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663795 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663797 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663800 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663803 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663806 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663808 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663811 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663814 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663817 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663820 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:41.665591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663824 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663829 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663834 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663837 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663842 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663845 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663848 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663852 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663855 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663857 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663860 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663863 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663866 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663869 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663872 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663875 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663878 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663881 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663884 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:41.666131 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663887 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663890 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663893 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663896 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663899 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663902 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663905 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663908 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663910 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663913 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663916 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663919 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663922 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663925 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663928 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663945 2552 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663949 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663953 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663958 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663961 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:41.666637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663964 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663966 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663969 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663973 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663976 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663978 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663981 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663984 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663987 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663989 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663992 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663995 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.663998 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664001 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664004 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664006 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664009 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664012 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664015 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664018 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:41.667187 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664021 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664024 2552 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664027 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664030 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664033 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664036 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664040 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664044 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664047 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664051 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664054 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664057 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664059 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664063 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664066 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664069 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664072 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664075 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664077 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:41.667733 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664081 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664083 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664493 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664498 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664501 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664504 2552 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664508 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664510 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664513 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664519 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664522 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664525 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664528 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664531 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664534 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664536 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664540 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664543 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664546 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664549 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:41.668274 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664551 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664554 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664557 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664560 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664563 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664566 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664569 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664572 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664574 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664577 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664580 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664583 2552 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664586 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664588 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664591 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664594 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664597 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664600 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664602 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664605 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:41.668807 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664608 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664612 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664615 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664618 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664621 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664624 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664626 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664630 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664637 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664641 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664644 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664647 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664650 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664653 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664655 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664659 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664661 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664664 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664667 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:41.669407 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664670 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664673 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664675 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664678 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664681 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664684 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664686 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664689 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664692 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664694 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664697 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664700 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664702 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664705 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664710 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664713 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664717 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664721 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664725 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:41.669917 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664729 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664732 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664737 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664741 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664743 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664746 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664749 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664752 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664755 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664758 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664761 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664764 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664767 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664770 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664773 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664777 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664780 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664782 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664785 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:41.670467 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664788 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664790 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664793 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.664796 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664869 2552 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664877 2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664883 2552 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664888 2552 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664895 2552 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664899 2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664903 2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664908 2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664911 2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664915 2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664918 2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664924 2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664927 2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664948 2552 flags.go:64] FLAG: --cgroup-root="" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664952 2552 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664956 2552 flags.go:64] FLAG: --client-ca-file="" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664959 2552 flags.go:64] FLAG: --cloud-provider="external" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664962 2552 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664967 2552 flags.go:64] FLAG: --cluster-domain="" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664971 2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 00:02:41.670984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664975 2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664978 2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664982 2552 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664986 2552 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664990 2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664994 2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.664997 2552 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665001 2552 flags.go:64] FLAG: --contention-profiling="false" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665004 2552 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665007 2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665011 2552 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665014 2552 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665018 2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665022 2552 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665025 2552 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665028 2552 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665033 2552 flags.go:64] FLAG: --enable-server="true" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665037 2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665042 2552 flags.go:64] FLAG: --event-burst="100" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665045 2552 flags.go:64] FLAG: --event-qps="50" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665048 2552 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665052 2552 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665055 2552 flags.go:64] FLAG: --eviction-hard="" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665061 2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 00:02:41.671709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665064 2552 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665068 2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665072 2552 flags.go:64] FLAG: --eviction-soft="" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665075 2552 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665078 2552 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665081 2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665084 2552 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665088 2552 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665091 2552 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665095 2552 flags.go:64] FLAG: --feature-gates="" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665099 2552 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665102 2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665105 2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665109 2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665112 2552 flags.go:64] FLAG: --healthz-port="10248" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665115 2552 flags.go:64] FLAG: --help="false" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665118 2552 flags.go:64] FLAG: --hostname-override="ip-10-0-132-168.ec2.internal" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665121 2552 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665125 2552 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665128 2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665132 2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665135 2552 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665138 2552 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 00:02:41.672670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665142 2552 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665146 2552 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665149 2552 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665153 2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665156 2552 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665159 2552 flags.go:64] FLAG: --kube-reserved="" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665162 2552 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665165 2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665170 2552 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665173 2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665177 2552 flags.go:64] FLAG: --lock-file="" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665180 2552 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665183 2552 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665186 2552 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665192 2552 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665195 2552 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665198 2552 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665202 2552 flags.go:64] FLAG: --logging-format="text" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665205 2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665209 2552 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665212 2552 flags.go:64] FLAG: --manifest-url="" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665215 2552 flags.go:64] FLAG: --manifest-url-header="" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665225 2552 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665228 2552 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665233 2552 flags.go:64] FLAG: --max-pods="110" Aug 04 00:02:41.673373 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665236 2552 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665239 2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665243 2552 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665246 2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665249 2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665252 2552 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665255 2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665263 2552 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665268 2552 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665271 2552 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665275 2552 flags.go:64] FLAG: --pod-cidr="" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665278 2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665284 2552 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665287 2552 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665290 2552 flags.go:64] FLAG: --pods-per-core="0" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665294 2552 flags.go:64] FLAG: --port="10250" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665299 2552 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665302 2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c269a4444eec30f6" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665306 2552 flags.go:64] FLAG: --qos-reserved="" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665309 2552 flags.go:64] FLAG: --read-only-port="10255" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665312 2552 flags.go:64] FLAG: --register-node="true" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665315 2552 flags.go:64] FLAG: --register-with-taints="" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665319 2552 flags.go:64] FLAG: --registry-burst="10" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665323 2552 flags.go:64] FLAG: --registry-qps="5" Aug 04 00:02:41.674080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665326 2552 flags.go:64] FLAG: --reserved-cpus="" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665329 2552 flags.go:64] FLAG: --reserved-memory="" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665333 2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665337 2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665341 2552 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665344 2552 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665347 2552 flags.go:64] FLAG: --runonce="false" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665350 2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665354 2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665357 2552 flags.go:64] FLAG: --seccomp-default="false" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665360 2552 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665363 2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665366 2552 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665370 2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665373 2552 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665376 2552 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665381 2552 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665384 2552 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665387 2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665391 2552 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665394 2552 flags.go:64] FLAG: --system-cgroups="" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665397 2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665403 2552 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665406 2552 flags.go:64] FLAG: --tls-cert-file="" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665411 2552 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 00:02:41.674745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665417 2552 flags.go:64] FLAG: --tls-min-version="" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665420 2552 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665423 2552 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665426 2552 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665430 2552 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665433 2552 flags.go:64] FLAG: --v="2" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665437 2552 flags.go:64] FLAG: --version="false" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665442 2552 flags.go:64] FLAG: --vmodule="" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665446 2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665449 2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665572 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665578 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665581 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665584 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665588 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665591 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665594 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665598 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665601 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665604 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665607 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665611 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:41.675483 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665614 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665617 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665620 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665623 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665627 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665631 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665634 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665638 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665641 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665646 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665649 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665652 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665656 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665659 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665662 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665665 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665668 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665670 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665673 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:41.676089 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665676 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665679 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665682 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665686 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665688 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665691 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665694 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665697 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665700 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665703 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665706 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665709 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665711 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665714 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665718 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665721 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665724 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665727 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665729 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665732 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:41.676591 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665735 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665738 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665742 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665745 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665748 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665752 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665756 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665759 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665763 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665765 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665768 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665771 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665774 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665776 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665779 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665782 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665785 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665788 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665791 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:41.677143 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665794 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665796 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665799 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665802 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665804 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665807 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665810 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665813 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665816 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665818 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665821 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665824 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665827 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665830 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665832 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665837 2552 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665840 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665843 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665846 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665848 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:41.677647 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665851 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665854 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665856 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665860 2552 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665862 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665866 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665870 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665874 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.665877 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.665883 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.674635 2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.674652 2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.675664 2552 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.675672 2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 00:02:41.678190 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675737 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675743 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675747 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675751 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675754 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675757 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675760 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675764 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675766 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675770 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675773 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675776 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675778 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675781 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675784 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675787 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675790 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675793 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675795 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675798 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:41.678576 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675801 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675811 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675815 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675818 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675821 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675824 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675827 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675829 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675834 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675838 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675841 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675844 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675847 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675851 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675854 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675857 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675859 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675863 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675866 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:41.679221 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675869 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675882 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675885 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675888 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675891 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675894 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675897 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675901 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675904 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675907 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675909 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675912 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675915 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675918 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675921 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675924 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675927 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675932 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675937 2552 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675941 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:41.679725 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675946 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675949 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675954 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675958 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675962 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675965 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675969 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675972 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675975 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675979 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675981 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675984 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675987 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675990 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675993 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675996 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.675999 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676003 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676005 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676008 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:41.680269 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676012 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676015 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676018 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676021 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676024 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676027 2552 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676031 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676033 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676036 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676041 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676044 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676047 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676051 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676054 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676058 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676061 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676064 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676067 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676070 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:41.680801 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676073 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.676078 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676204 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676210 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676213 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676217 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676220 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676224 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676227 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676230 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676233 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676236 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676239 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676242 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676246 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:41.681353 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676248 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676252 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676254 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676257 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676260 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676263 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676266 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676269 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676272 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676274 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676277 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676280 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676283 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676286 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676289 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676292 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676295 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676298 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676301 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676303 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:41.681765 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676306 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676309 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676312 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676315 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676318 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676321 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676323 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676326 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676329 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676332 2552 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676335 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676337 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676340 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676343 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676346 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676349 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676353 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676356 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676358 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676361 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:41.682324 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676364 2552 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676367 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676370 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676373 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676376 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676378 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676381 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676384 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676387 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676391 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676395 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676398 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676402 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676405 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676408 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676411 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676414 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676417 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676420 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:41.682857 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676423 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676426 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676429 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676431 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676434 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676437 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676441 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676444 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676447 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676449 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676452 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676455 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676458 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676460 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676463 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676467 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676471 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676474 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676477 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:41.683484 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676480 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676483 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676485 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676489 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676493 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676496 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676499 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:41.676502 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.676508 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.676638 2552 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.680926 2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.682813 2552 server.go:1019] "Starting client certificate rotation" Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.682985 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 00:02:41.684002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.683043 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 00:02:41.686875 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.686856 2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 00:02:41.692578 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.692557 2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 00:02:41.703556 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.703540 2552 log.go:25] "Validated CRI v1 runtime API" Aug 04 00:02:41.708201 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.708183 2552 log.go:25] "Validated CRI v1 image API" Aug 04 00:02:41.710200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.710179 2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 00:02:41.714367 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.714347 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 00:02:41.715175 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.715143 2552 fs.go:135] Filesystem UUIDs: map[7499c465-5a23-4cbe-9732-0d45134012b3:/dev/nvme0n1p3 79d932fd-d027-4e8f-8ae5-b99ab77936a3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Aug 04 00:02:41.715246 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.715174 2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 00:02:41.721725 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.721608 2552 manager.go:216] Machine: {Timestamp:2026-08-04 00:02:41.721096527 +0000 UTC m=+0.384006005 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105333 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9e5dc5629dcb260eacefea522b43 SystemUUID:ec2a9e5d-c562-9dcb-260e-acefea522b43 BootID:9a24ddcd-a29e-4add-a878-259c1ba8754b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3b:b2:a7:ac:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3b:b2:a7:ac:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:b7:0f:6c:6d:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 00:02:41.721725 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.721722 2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 00:02:41.721873 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.721807 2552 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 00:02:41.725476 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.725450 2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 00:02:41.725619 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.725479 2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-168.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 00:02:41.726784 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.726773 2552 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 00:02:41.726830 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.726787 2552 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 00:02:41.726830 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.726795 2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 00:02:41.726830 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.726809 2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 00:02:41.728274 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.728263 2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 00:02:41.733467 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.733454 2552 state_mem.go:36] "Initialized new in-memory state store" Aug 04 00:02:41.733626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.733615 2552 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 00:02:41.733658 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.733647 2552 kubelet.go:486] "Attempting to sync node with API server" Aug 04 00:02:41.733658 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.733656 2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 00:02:41.733721 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.733674 2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 00:02:41.733721 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.733683 2552 kubelet.go:398] "Adding apiserver pod source" Aug 04 00:02:41.733721 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.733692 2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 00:02:41.734055 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.734043 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 00:02:41.734117 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.734063 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 00:02:41.736754 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.736737 2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 00:02:41.737895 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.737877 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4qgjr" Aug 04 00:02:41.739459 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.739445 2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 00:02:41.739500 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.739485 2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 00:02:41.740235 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740223 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 00:02:41.740268 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740241 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 00:02:41.740268 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740248 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 00:02:41.740268 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740254 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 00:02:41.740268 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740262 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 00:02:41.740386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740270 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 00:02:41.740386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740284 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 00:02:41.740386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740294 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 00:02:41.740386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740302 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 00:02:41.740386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740308 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 00:02:41.740386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740323 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 00:02:41.740386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.740329 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 00:02:41.741237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.741221 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 00:02:41.741237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.741235 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 00:02:41.743523 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.743495 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 00:02:41.743702 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.743682 2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-168.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 00:02:41.743761 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.743742 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 00:02:41.745038 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.745022 2552 server.go:1268] "Started kubelet" Aug 04 00:02:41.745129 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.745098 2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 00:02:41.745929 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.745863 2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 00:02:41.746015 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.745959 2552 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 00:02:41.746038 ip-10-0-132-168 systemd[1]: Started Kubernetes Kubelet. Aug 04 00:02:41.747709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.747697 2552 server.go:310] "Adding debug handlers to kubelet server" Aug 04 00:02:41.748325 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.748308 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4qgjr" Aug 04 00:02:41.749392 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.749371 2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 00:02:41.752389 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.750616 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-168.ec2.internal.18c8713da013c128 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-168.ec2.internal,UID:ip-10-0-132-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-168.ec2.internal,},FirstTimestamp:2026-08-04 00:02:41.7449946 +0000 UTC m=+0.407904071,LastTimestamp:2026-08-04 00:02:41.7449946 +0000 UTC m=+0.407904071,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-168.ec2.internal,}" Aug 04 00:02:41.752797 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.752780 2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 00:02:41.756637 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.756618 2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 00:02:41.756637 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.756630 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 00:02:41.756785 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.756727 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:41.757255 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757233 2552 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 00:02:41.757255 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757257 2552 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 00:02:41.757388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757368 2552 reconstruct.go:97] "Volume reconstruction finished" Aug 04 00:02:41.757388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757380 2552 reconciler.go:29] "Reconciler: start to sync state" Aug 04 00:02:41.757388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757367 2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 00:02:41.757769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757752 2552 factory.go:153] Registering CRI-O factory Aug 04 00:02:41.757769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757770 2552 factory.go:223] Registration of the crio container factory successfully Aug 04 00:02:41.757924 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757825 2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 00:02:41.757924 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757835 2552 factory.go:55] Registering systemd factory Aug 04 00:02:41.757924 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757843 2552 factory.go:223] Registration of the systemd container factory successfully Aug 04 00:02:41.757924 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757868 2552 factory.go:103] Registering Raw factory Aug 04 00:02:41.757924 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.757897 2552 manager.go:1196] Started watching for new ooms in manager Aug 04 00:02:41.758550 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.758537 2552 manager.go:349] Starting recovery of all containers Aug 04 00:02:41.759362 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.759342 2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:41.766187 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.766147 2552 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-168.ec2.internal\" not found" node="ip-10-0-132-168.ec2.internal" Aug 04 00:02:41.769192 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.769177 2552 manager.go:354] Recovery completed Aug 04 00:02:41.770618 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.770505 2552 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Aug 04 00:02:41.773678 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.773665 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:41.775626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.775609 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:41.775707 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.775642 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:41.775707 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.775658 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:41.776247 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.776229 2552 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 00:02:41.776247 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.776245 2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 00:02:41.776350 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.776262 2552 state_mem.go:36] "Initialized new in-memory state store" Aug 04 00:02:41.776350 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.776237 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:41.777108 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.777088 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:41.777183 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.777118 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:41.777183 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.777128 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:41.778462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.778450 2552 policy_none.go:49] "None policy: Start" Aug 04 00:02:41.778514 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.778466 2552 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 00:02:41.778514 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.778478 2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 00:02:41.780483 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.780472 2552 policy_none.go:47] "Start" Aug 04 00:02:41.824130 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.824111 2552 manager.go:337] "Starting Device Plugin manager" Aug 04 00:02:41.824232 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.824148 2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 00:02:41.824232 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.824158 2552 server.go:87] "Starting device plugin registration server" Aug 04 00:02:41.824429 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.824410 2552 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 00:02:41.824524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.824428 2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 00:02:41.824524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.824500 2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 00:02:41.824635 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.824584 2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 00:02:41.824635 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.824591 2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 00:02:41.825192 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.825170 2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 00:02:41.825276 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.825228 2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:41.927184 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.927134 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:41.929501 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.928236 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:41.929501 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.928275 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:41.929501 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.928288 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:41.929501 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.928317 2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-168.ec2.internal" Aug 04 00:02:41.938089 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.938073 2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-168.ec2.internal" Aug 04 00:02:41.938139 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.938097 2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-168.ec2.internal\": node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:41.945269 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.945238 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 00:02:41.946463 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.946445 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 00:02:41.946569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.946487 2552 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 00:02:41.946569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.946511 2552 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 00:02:41.946569 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.946543 2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 00:02:41.949152 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:41.949128 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:41.952368 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:41.952344 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.046839 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.046794 2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"] Aug 04 00:02:42.046996 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.046910 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:42.047932 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.047912 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:42.048036 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.047943 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:42.048036 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.047953 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:42.050372 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.050360 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:42.050473 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.050459 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.050515 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.050491 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:42.051308 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.051288 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:42.051308 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.051307 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:42.051457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.051317 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:42.051457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.051328 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:42.051457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.051329 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:42.051457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.051340 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:42.053258 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.053244 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.053670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.053657 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.053720 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.053683 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:42.054446 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.054429 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:42.054524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.054461 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:42.054524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.054477 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:42.059034 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.059013 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01b58fa67d743aa84130ed84e63b60ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"01b58fa67d743aa84130ed84e63b60ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.059123 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.059039 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01b58fa67d743aa84130ed84e63b60ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"01b58fa67d743aa84130ed84e63b60ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.059123 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.059059 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c14b132a47c877cb6e709b2d262ede1-config\") pod \"kube-apiserver-proxy-ip-10-0-132-168.ec2.internal\" (UID: \"7c14b132a47c877cb6e709b2d262ede1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.080689 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.080672 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-168.ec2.internal\" not found" node="ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.085555 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.085539 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-168.ec2.internal\" not found" node="ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.153464 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.153443 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.159382 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.159366 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01b58fa67d743aa84130ed84e63b60ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"01b58fa67d743aa84130ed84e63b60ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.159444 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.159391 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c14b132a47c877cb6e709b2d262ede1-config\") pod \"kube-apiserver-proxy-ip-10-0-132-168.ec2.internal\" (UID: \"7c14b132a47c877cb6e709b2d262ede1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.159444 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.159408 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01b58fa67d743aa84130ed84e63b60ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"01b58fa67d743aa84130ed84e63b60ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.159514 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.159451 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01b58fa67d743aa84130ed84e63b60ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"01b58fa67d743aa84130ed84e63b60ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.159514 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.159467 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01b58fa67d743aa84130ed84e63b60ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"01b58fa67d743aa84130ed84e63b60ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.159514 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.159486 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c14b132a47c877cb6e709b2d262ede1-config\") pod \"kube-apiserver-proxy-ip-10-0-132-168.ec2.internal\" (UID: \"7c14b132a47c877cb6e709b2d262ede1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.254074 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.254003 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.361744 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.361708 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.383491 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.383465 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.389776 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.389439 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.461809 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.461766 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.562907 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.562835 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.663773 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.663732 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.681696 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.681662 2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 00:02:42.681848 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.681829 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 00:02:42.681909 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.681847 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 00:02:42.734754 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.734725 2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:42.751775 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.751744 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-02 23:57:41 +0000 UTC" deadline="2028-04-06 13:34:35.536196968 +0000 UTC" Aug 04 00:02:42.751775 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.751774 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14677h31m52.784428073s" Aug 04 00:02:42.764104 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.764079 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.864409 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:42.864340 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Aug 04 00:02:42.941296 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.941273 2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:42.957891 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.957869 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.969667 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.969649 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 00:02:42.970764 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.970748 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Aug 04 00:02:42.979339 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:42.979320 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 00:02:43.016988 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:43.016953 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c14b132a47c877cb6e709b2d262ede1.slice/crio-24a73f8f2108880afc08266fb20975d8dd39181e50a6f42fc91dda54f99312fc WatchSource:0}: Error finding container 24a73f8f2108880afc08266fb20975d8dd39181e50a6f42fc91dda54f99312fc: Status 404 returned error can't find the container with id 24a73f8f2108880afc08266fb20975d8dd39181e50a6f42fc91dda54f99312fc Aug 04 00:02:43.024686 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.024668 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:02:43.028530 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:43.028507 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b58fa67d743aa84130ed84e63b60ea.slice/crio-2033a03dcc1be79ff8ac863e15cd886b931a561f4695ad1dfec57020d5336fd8 WatchSource:0}: Error finding container 2033a03dcc1be79ff8ac863e15cd886b931a561f4695ad1dfec57020d5336fd8: Status 404 returned error can't find the container with id 2033a03dcc1be79ff8ac863e15cd886b931a561f4695ad1dfec57020d5336fd8 Aug 04 00:02:43.735432 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.735400 2552 apiserver.go:52] "Watching apiserver" Aug 04 00:02:43.747112 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.747081 2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 00:02:43.748578 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.748537 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-nhd8z","kube-system/konnectivity-agent-8vz67","kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc","openshift-cluster-node-tuning-operator/tuned-mprw6","openshift-dns/node-resolver-v8btt","openshift-multus/multus-additional-cni-plugins-cqmnm","openshift-ovn-kubernetes/ovnkube-node-r7tkg","openshift-image-registry/node-ca-fkmvm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal","openshift-multus/multus-4wxgh","openshift-multus/network-metrics-daemon-zwpn4","openshift-network-diagnostics/network-check-target-kwjfz","openshift-network-operator/iptables-alerter-5fnhn"] Aug 04 00:02:43.751245 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.751221 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.753749 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.753637 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:43.753749 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.753710 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:02:43.754149 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.754064 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 00:02:43.754149 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.754069 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 00:02:43.754298 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.754282 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 00:02:43.754602 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.754581 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 00:02:43.754693 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.754604 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2sdw4\"" Aug 04 00:02:43.754757 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.754587 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 00:02:43.756172 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.756100 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.758081 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.758062 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 00:02:43.758828 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.758619 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.759136 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.759114 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 00:02:43.759229 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.759163 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-scmlx\"" Aug 04 00:02:43.759300 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.759199 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:02:43.759562 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.759537 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 00:02:43.761683 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.761539 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.761683 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.761573 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 00:02:43.761841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.761685 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-shf2g\"" Aug 04 00:02:43.762083 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.762064 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 00:02:43.762083 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.762076 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 00:02:43.763840 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.763813 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:02:43.764037 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.764019 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:43.764333 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.764267 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 00:02:43.764333 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.764286 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vs7sl\"" Aug 04 00:02:43.766848 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.766419 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.766848 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.766644 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 00:02:43.766848 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.766794 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqbpk\"" Aug 04 00:02:43.767137 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.767122 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768133 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768169 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768200 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrhl\" (UniqueName: \"kubernetes.io/projected/3404f652-a2a1-46b1-9130-a67db57919f0-kube-api-access-hhrhl\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768225 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-os-release\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768250 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a966195c-aac5-44da-9599-8bad5ef0e2c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768301 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-device-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768369 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-system-cni-dir\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768430 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-cnibin\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768479 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzdk\" (UniqueName: \"kubernetes.io/projected/a966195c-aac5-44da-9599-8bad5ef0e2c3-kube-api-access-chzdk\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768512 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txz7d\" (UniqueName: \"kubernetes.io/projected/e2b0ffc0-bd27-4fac-af16-724ba66c64de-kube-api-access-txz7d\") pod \"iptables-alerter-5fnhn\" (UID: \"e2b0ffc0-bd27-4fac-af16-724ba66c64de\") " pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768540 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768549 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768580 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-socket-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768625 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-registration-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768653 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-sys-fs\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.769095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768680 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a966195c-aac5-44da-9599-8bad5ef0e2c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.770061 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768749 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a966195c-aac5-44da-9599-8bad5ef0e2c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.770061 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768814 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:43.770061 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768851 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2b0ffc0-bd27-4fac-af16-724ba66c64de-iptables-alerter-script\") pod \"iptables-alerter-5fnhn\" (UID: \"e2b0ffc0-bd27-4fac-af16-724ba66c64de\") " pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.770061 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.768880 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2b0ffc0-bd27-4fac-af16-724ba66c64de-host-slash\") pod \"iptables-alerter-5fnhn\" (UID: \"e2b0ffc0-bd27-4fac-af16-724ba66c64de\") " pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.770061 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.769296 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-76cpt\"" Aug 04 00:02:43.770061 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.769324 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 00:02:43.770061 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.769346 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 00:02:43.770061 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.769406 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:43.770061 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.769479 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:02:43.770964 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.770910 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 00:02:43.771690 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.771610 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:43.774428 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.774230 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 00:02:43.774575 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.774558 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.775599 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.775578 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-plwr8\"" Aug 04 00:02:43.775916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.775895 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 00:02:43.777731 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.777058 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 00:02:43.777731 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.777283 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 00:02:43.777731 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.777355 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lgxcd\"" Aug 04 00:02:43.777731 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.777480 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 00:02:43.777731 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.777592 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 00:02:43.778062 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.777812 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 00:02:43.779116 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.778989 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 00:02:43.779708 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.779318 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.784448 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.784273 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:43.784448 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.784322 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 00:02:43.784448 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.784347 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:02:43.785216 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.785192 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zw9fr\"" Aug 04 00:02:43.787481 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.787383 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:43.796713 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.796691 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c46nv" Aug 04 00:02:43.810041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.810014 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c46nv" Aug 04 00:02:43.859135 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.859103 2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 00:02:43.869601 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869573 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2b0ffc0-bd27-4fac-af16-724ba66c64de-iptables-alerter-script\") pod \"iptables-alerter-5fnhn\" (UID: \"e2b0ffc0-bd27-4fac-af16-724ba66c64de\") " pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.869796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869619 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-system-cni-dir\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.869796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869656 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a966195c-aac5-44da-9599-8bad5ef0e2c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.869796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869686 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a77bedb-69cf-4cd2-8a73-79829ed0c663-konnectivity-ca\") pod \"konnectivity-agent-8vz67\" (UID: \"9a77bedb-69cf-4cd2-8a73-79829ed0c663\") " pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:43.869796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869711 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-run-netns\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.869796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869734 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-cni-bin\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.869796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869760 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-multus-socket-dir-parent\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.869796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869784 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-etc-kubernetes\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869811 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-cnibin\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869833 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-multus-cni-dir\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869881 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-var-lib-kubelet\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869926 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869960 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-etc-openvswitch\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.869985 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-run-openvswitch\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870011 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6725222-fca9-49d7-b2c5-9b172b300071-multus-daemon-config\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870043 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870067 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6920ae2c-7e06-47e6-9101-47f93c1b3226-tmp-dir\") pod \"node-resolver-v8btt\" (UID: \"6920ae2c-7e06-47e6-9101-47f93c1b3226\") " pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870113 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc72d8b6-1df2-48fb-b76e-671a7c4b905e-serviceca\") pod \"node-ca-fkmvm\" (UID: \"cc72d8b6-1df2-48fb-b76e-671a7c4b905e\") " pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870138 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2rrz\" (UniqueName: \"kubernetes.io/projected/cc72d8b6-1df2-48fb-b76e-671a7c4b905e-kube-api-access-k2rrz\") pod \"node-ca-fkmvm\" (UID: \"cc72d8b6-1df2-48fb-b76e-671a7c4b905e\") " pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.870200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870174 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4179408d-c308-41a7-b4a1-7446d7cd2a35-ovnkube-config\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870203 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4179408d-c308-41a7-b4a1-7446d7cd2a35-ovn-node-metrics-cert\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870235 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-hostroot\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870258 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870285 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4xr\" (UniqueName: \"kubernetes.io/projected/4179408d-c308-41a7-b4a1-7446d7cd2a35-kube-api-access-lc4xr\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870310 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-os-release\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870335 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96rsp\" (UniqueName: \"kubernetes.io/projected/f6725222-fca9-49d7-b2c5-9b172b300071-kube-api-access-96rsp\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870354 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2b0ffc0-bd27-4fac-af16-724ba66c64de-iptables-alerter-script\") pod \"iptables-alerter-5fnhn\" (UID: \"e2b0ffc0-bd27-4fac-af16-724ba66c64de\") " pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870413 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-device-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870444 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-system-cni-dir\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870467 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870472 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc72d8b6-1df2-48fb-b76e-671a7c4b905e-host\") pod \"node-ca-fkmvm\" (UID: \"cc72d8b6-1df2-48fb-b76e-671a7c4b905e\") " pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870504 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-kubelet\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870511 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-cnibin\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870584 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-slash\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870606 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a966195c-aac5-44da-9599-8bad5ef0e2c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.870769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870615 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-run-ovn\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870667 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870686 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-device-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870691 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870701 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4179408d-c308-41a7-b4a1-7446d7cd2a35-ovnkube-script-lib\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870735 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-system-cni-dir\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870738 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-tmp\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870777 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-kubelet-config\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870809 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870848 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-socket-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870902 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-sysconfig\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870940 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-var-lib-kubelet\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870960 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-socket-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.870974 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-node-log\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871002 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871037 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2b0ffc0-bd27-4fac-af16-724ba66c64de-host-slash\") pod \"iptables-alerter-5fnhn\" (UID: \"e2b0ffc0-bd27-4fac-af16-724ba66c64de\") " pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.871549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871062 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871080 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-tuned\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871103 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-run-netns\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871121 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-sysctl-d\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871137 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-systemd\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871158 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-dbus\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871174 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-multus-conf-dir\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871191 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzdk\" (UniqueName: \"kubernetes.io/projected/a966195c-aac5-44da-9599-8bad5ef0e2c3-kube-api-access-chzdk\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871208 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6920ae2c-7e06-47e6-9101-47f93c1b3226-hosts-file\") pod \"node-resolver-v8btt\" (UID: \"6920ae2c-7e06-47e6-9101-47f93c1b3226\") " pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871223 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a77bedb-69cf-4cd2-8a73-79829ed0c663-agent-certs\") pod \"konnectivity-agent-8vz67\" (UID: \"9a77bedb-69cf-4cd2-8a73-79829ed0c663\") " pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871245 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tphqg\" (UniqueName: \"kubernetes.io/projected/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-kube-api-access-tphqg\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871269 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-systemd-units\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871313 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-log-socket\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871334 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871338 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2b0ffc0-bd27-4fac-af16-724ba66c64de-host-slash\") pod \"iptables-alerter-5fnhn\" (UID: \"e2b0ffc0-bd27-4fac-af16-724ba66c64de\") " pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871352 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6725222-fca9-49d7-b2c5-9b172b300071-cni-binary-copy\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871395 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a966195c-aac5-44da-9599-8bad5ef0e2c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.872393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871434 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-kubernetes\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871444 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871461 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-run-systemd\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871647 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-run-k8s-cni-cncf-io\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871711 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-var-lib-cni-bin\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871739 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-var-lib-cni-multus\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871762 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dphn\" (UniqueName: \"kubernetes.io/projected/6920ae2c-7e06-47e6-9101-47f93c1b3226-kube-api-access-5dphn\") pod \"node-resolver-v8btt\" (UID: \"6920ae2c-7e06-47e6-9101-47f93c1b3226\") " pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871794 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-run\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871815 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-var-lib-openvswitch\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871841 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-cni-netd\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871865 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-cnibin\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871928 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-run-multus-certs\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871992 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a966195c-aac5-44da-9599-8bad5ef0e2c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.871996 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrhl\" (UniqueName: \"kubernetes.io/projected/3404f652-a2a1-46b1-9130-a67db57919f0-kube-api-access-hhrhl\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872239 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-os-release\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872285 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-modprobe-d\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872312 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-sysctl-conf\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.873272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872345 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-sys\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872370 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9p2r\" (UniqueName: \"kubernetes.io/projected/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-kube-api-access-g9p2r\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872394 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-host\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872407 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a966195c-aac5-44da-9599-8bad5ef0e2c3-os-release\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872433 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txz7d\" (UniqueName: \"kubernetes.io/projected/e2b0ffc0-bd27-4fac-af16-724ba66c64de-kube-api-access-txz7d\") pod \"iptables-alerter-5fnhn\" (UID: \"e2b0ffc0-bd27-4fac-af16-724ba66c64de\") " pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872464 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-registration-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872492 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-sys-fs\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872520 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a966195c-aac5-44da-9599-8bad5ef0e2c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872562 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-lib-modules\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872590 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4179408d-c308-41a7-b4a1-7446d7cd2a35-env-overrides\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872684 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-sys-fs\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.872692 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3404f652-a2a1-46b1-9130-a67db57919f0-registration-dir\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.874171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.873187 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a966195c-aac5-44da-9599-8bad5ef0e2c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.876764 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.876747 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:43.876875 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.876770 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:43.876875 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.876787 2552 projected.go:196] Error preparing data for projected volume kube-api-access-gz9jd for pod openshift-network-diagnostics/network-check-target-kwjfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:43.876875 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.876854 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd podName:f81f2275-73c2-4b81-8589-ef82bf8cf6ca nodeName:}" failed. No retries permitted until 2026-08-04 00:02:44.376833443 +0000 UTC m=+3.039742907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gz9jd" (UniqueName: "kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd") pod "network-check-target-kwjfz" (UID: "f81f2275-73c2-4b81-8589-ef82bf8cf6ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:43.878285 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.877891 2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 00:02:43.883410 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.883067 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrhl\" (UniqueName: \"kubernetes.io/projected/3404f652-a2a1-46b1-9130-a67db57919f0-kube-api-access-hhrhl\") pod \"aws-ebs-csi-driver-node-wx2mc\" (UID: \"3404f652-a2a1-46b1-9130-a67db57919f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:43.883410 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.883105 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txz7d\" (UniqueName: \"kubernetes.io/projected/e2b0ffc0-bd27-4fac-af16-724ba66c64de-kube-api-access-txz7d\") pod \"iptables-alerter-5fnhn\" (UID: \"e2b0ffc0-bd27-4fac-af16-724ba66c64de\") " pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:43.883800 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.883777 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzdk\" (UniqueName: \"kubernetes.io/projected/a966195c-aac5-44da-9599-8bad5ef0e2c3-kube-api-access-chzdk\") pod \"multus-additional-cni-plugins-cqmnm\" (UID: \"a966195c-aac5-44da-9599-8bad5ef0e2c3\") " pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:43.952476 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.952414 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"01b58fa67d743aa84130ed84e63b60ea","Type":"ContainerStarted","Data":"2033a03dcc1be79ff8ac863e15cd886b931a561f4695ad1dfec57020d5336fd8"} Aug 04 00:02:43.953903 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.953853 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" event={"ID":"7c14b132a47c877cb6e709b2d262ede1","Type":"ContainerStarted","Data":"24a73f8f2108880afc08266fb20975d8dd39181e50a6f42fc91dda54f99312fc"} Aug 04 00:02:43.972891 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.972855 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tphqg\" (UniqueName: \"kubernetes.io/projected/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-kube-api-access-tphqg\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:43.973041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.972896 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-systemd-units\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.973041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.972922 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-log-socket\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.973041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.972962 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.973041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.972987 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6725222-fca9-49d7-b2c5-9b172b300071-cni-binary-copy\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.973041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973015 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-kubernetes\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.973041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973033 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-run-systemd\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973051 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-run-k8s-cni-cncf-io\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973068 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-var-lib-cni-bin\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973089 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-var-lib-cni-multus\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973111 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dphn\" (UniqueName: \"kubernetes.io/projected/6920ae2c-7e06-47e6-9101-47f93c1b3226-kube-api-access-5dphn\") pod \"node-resolver-v8btt\" (UID: \"6920ae2c-7e06-47e6-9101-47f93c1b3226\") " pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973136 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-run\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973158 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-var-lib-openvswitch\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973172 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-kubernetes\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973206 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-cni-netd\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973226 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-cnibin\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973234 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-systemd-units\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973252 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-run-multus-certs\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973278 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-modprobe-d\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973278 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-log-socket\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973295 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-sysctl-conf\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973312 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-sys\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973321 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973328 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9p2r\" (UniqueName: \"kubernetes.io/projected/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-kube-api-access-g9p2r\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.973359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973346 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-host\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973370 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-lib-modules\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973388 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4179408d-c308-41a7-b4a1-7446d7cd2a35-env-overrides\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973405 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-system-cni-dir\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973422 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a77bedb-69cf-4cd2-8a73-79829ed0c663-konnectivity-ca\") pod \"konnectivity-agent-8vz67\" (UID: \"9a77bedb-69cf-4cd2-8a73-79829ed0c663\") " pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973448 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-run-netns\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973474 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-cni-bin\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973494 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-multus-socket-dir-parent\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973510 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-etc-kubernetes\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973532 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-multus-cni-dir\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973549 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-var-lib-kubelet\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973566 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-etc-openvswitch\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973582 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-run-openvswitch\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973600 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6725222-fca9-49d7-b2c5-9b172b300071-multus-daemon-config\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973624 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6920ae2c-7e06-47e6-9101-47f93c1b3226-tmp-dir\") pod \"node-resolver-v8btt\" (UID: \"6920ae2c-7e06-47e6-9101-47f93c1b3226\") " pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973655 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc72d8b6-1df2-48fb-b76e-671a7c4b905e-serviceca\") pod \"node-ca-fkmvm\" (UID: \"cc72d8b6-1df2-48fb-b76e-671a7c4b905e\") " pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973670 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2rrz\" (UniqueName: \"kubernetes.io/projected/cc72d8b6-1df2-48fb-b76e-671a7c4b905e-kube-api-access-k2rrz\") pod \"node-ca-fkmvm\" (UID: \"cc72d8b6-1df2-48fb-b76e-671a7c4b905e\") " pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973694 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4179408d-c308-41a7-b4a1-7446d7cd2a35-ovnkube-config\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.974204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973723 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4179408d-c308-41a7-b4a1-7446d7cd2a35-ovn-node-metrics-cert\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973742 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-hostroot\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973760 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973781 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4xr\" (UniqueName: \"kubernetes.io/projected/4179408d-c308-41a7-b4a1-7446d7cd2a35-kube-api-access-lc4xr\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973802 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-os-release\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973817 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96rsp\" (UniqueName: \"kubernetes.io/projected/f6725222-fca9-49d7-b2c5-9b172b300071-kube-api-access-96rsp\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973835 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc72d8b6-1df2-48fb-b76e-671a7c4b905e-host\") pod \"node-ca-fkmvm\" (UID: \"cc72d8b6-1df2-48fb-b76e-671a7c4b905e\") " pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973851 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-kubelet\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973874 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-slash\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973893 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-run-ovn\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973902 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6725222-fca9-49d7-b2c5-9b172b300071-cni-binary-copy\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973909 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973977 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4179408d-c308-41a7-b4a1-7446d7cd2a35-ovnkube-script-lib\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973990 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.973993 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-cni-bin\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974010 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-tmp\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974062 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-multus-socket-dir-parent\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974062 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-var-lib-cni-bin\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974101 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-var-lib-cni-multus\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974116 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-etc-kubernetes\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974167 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-var-lib-openvswitch\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974209 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-cni-netd\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974263 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-cnibin\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974305 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-run-multus-certs\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974430 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-modprobe-d\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974445 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-multus-cni-dir\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974488 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-var-lib-kubelet\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974523 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-etc-openvswitch\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974551 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-sysctl-conf\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974565 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-run-openvswitch\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974604 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-sys\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974651 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-run-k8s-cni-cncf-io\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.974925 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-host\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.975067 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6725222-fca9-49d7-b2c5-9b172b300071-multus-daemon-config\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.975076 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-lib-modules\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.975347 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6920ae2c-7e06-47e6-9101-47f93c1b3226-tmp-dir\") pod \"node-resolver-v8btt\" (UID: \"6920ae2c-7e06-47e6-9101-47f93c1b3226\") " pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:43.975899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.975529 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4179408d-c308-41a7-b4a1-7446d7cd2a35-env-overrides\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.975600 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-system-cni-dir\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.975761 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc72d8b6-1df2-48fb-b76e-671a7c4b905e-serviceca\") pod \"node-ca-fkmvm\" (UID: \"cc72d8b6-1df2-48fb-b76e-671a7c4b905e\") " pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976143 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a77bedb-69cf-4cd2-8a73-79829ed0c663-konnectivity-ca\") pod \"konnectivity-agent-8vz67\" (UID: \"9a77bedb-69cf-4cd2-8a73-79829ed0c663\") " pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976209 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-run-netns\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976429 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4179408d-c308-41a7-b4a1-7446d7cd2a35-ovnkube-config\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976666 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-tmp\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976696 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4179408d-c308-41a7-b4a1-7446d7cd2a35-ovnkube-script-lib\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976711 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-kubelet-config\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976736 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976756 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-sysconfig\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976764 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc72d8b6-1df2-48fb-b76e-671a7c4b905e-host\") pod \"node-ca-fkmvm\" (UID: \"cc72d8b6-1df2-48fb-b76e-671a7c4b905e\") " pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976772 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-var-lib-kubelet\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976798 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-node-log\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.976807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976810 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-hostroot\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976848 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-tuned\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.976855 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.976875 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976873 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-run-netns\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.976918 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret podName:c93f255c-fe84-43d8-8ebd-4c54a072e1f8 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:44.476900262 +0000 UTC m=+3.139809746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret") pod "global-pull-secret-syncer-nhd8z" (UID: "c93f255c-fe84-43d8-8ebd-4c54a072e1f8") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976973 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-sysconfig\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976976 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-sysctl-d\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977007 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-systemd\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:43.977050 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs podName:fc8dfa3c-04cf-4384-ab5f-6cbee7582434 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:44.47703542 +0000 UTC m=+3.139944893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs") pod "network-metrics-daemon-zwpn4" (UID: "fc8dfa3c-04cf-4384-ab5f-6cbee7582434") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.976929 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-host-run-netns\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977072 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-run-systemd\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977120 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-systemd\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977161 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-var-lib-kubelet\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977193 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-os-release\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977197 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-node-log\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977229 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-kubelet-config\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:43.977527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977235 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-dbus\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977267 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-run-ovn\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977328 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-kubelet\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977357 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-dbus\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977364 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4179408d-c308-41a7-b4a1-7446d7cd2a35-host-slash\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977414 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-multus-conf-dir\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977468 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6725222-fca9-49d7-b2c5-9b172b300071-multus-conf-dir\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977508 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6920ae2c-7e06-47e6-9101-47f93c1b3226-hosts-file\") pod \"node-resolver-v8btt\" (UID: \"6920ae2c-7e06-47e6-9101-47f93c1b3226\") " pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977509 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-run\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977470 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-sysctl-d\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977544 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a77bedb-69cf-4cd2-8a73-79829ed0c663-agent-certs\") pod \"konnectivity-agent-8vz67\" (UID: \"9a77bedb-69cf-4cd2-8a73-79829ed0c663\") " pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:43.978462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.977564 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6920ae2c-7e06-47e6-9101-47f93c1b3226-hosts-file\") pod \"node-resolver-v8btt\" (UID: \"6920ae2c-7e06-47e6-9101-47f93c1b3226\") " pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:43.984121 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.984096 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2rrz\" (UniqueName: \"kubernetes.io/projected/cc72d8b6-1df2-48fb-b76e-671a7c4b905e-kube-api-access-k2rrz\") pod \"node-ca-fkmvm\" (UID: \"cc72d8b6-1df2-48fb-b76e-671a7c4b905e\") " pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:43.987884 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.987701 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9p2r\" (UniqueName: \"kubernetes.io/projected/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-kube-api-access-g9p2r\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.990018 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.989225 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4179408d-c308-41a7-b4a1-7446d7cd2a35-ovn-node-metrics-cert\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.990855 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.990162 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef-etc-tuned\") pod \"tuned-mprw6\" (UID: \"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef\") " pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:43.992733 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.992324 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tphqg\" (UniqueName: \"kubernetes.io/projected/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-kube-api-access-tphqg\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:43.992733 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.992692 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a77bedb-69cf-4cd2-8a73-79829ed0c663-agent-certs\") pod \"konnectivity-agent-8vz67\" (UID: \"9a77bedb-69cf-4cd2-8a73-79829ed0c663\") " pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:43.992894 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.992816 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96rsp\" (UniqueName: \"kubernetes.io/projected/f6725222-fca9-49d7-b2c5-9b172b300071-kube-api-access-96rsp\") pod \"multus-4wxgh\" (UID: \"f6725222-fca9-49d7-b2c5-9b172b300071\") " pod="openshift-multus/multus-4wxgh" Aug 04 00:02:43.994102 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.994082 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4xr\" (UniqueName: \"kubernetes.io/projected/4179408d-c308-41a7-b4a1-7446d7cd2a35-kube-api-access-lc4xr\") pod \"ovnkube-node-r7tkg\" (UID: \"4179408d-c308-41a7-b4a1-7446d7cd2a35\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:43.994363 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:43.994345 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dphn\" (UniqueName: \"kubernetes.io/projected/6920ae2c-7e06-47e6-9101-47f93c1b3226-kube-api-access-5dphn\") pod \"node-resolver-v8btt\" (UID: \"6920ae2c-7e06-47e6-9101-47f93c1b3226\") " pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:44.065106 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.064754 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" Aug 04 00:02:44.074255 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.074232 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5fnhn" Aug 04 00:02:44.088514 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.088488 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" Aug 04 00:02:44.095675 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.095651 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mprw6" Aug 04 00:02:44.102619 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.102599 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v8btt" Aug 04 00:02:44.112444 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.112424 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fkmvm" Aug 04 00:02:44.119279 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.119259 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:44.128117 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.128096 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:02:44.134002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.133979 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4wxgh" Aug 04 00:02:44.218238 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.218185 2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:44.253145 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:44.253112 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b0ffc0_bd27_4fac_af16_724ba66c64de.slice/crio-c0e344b95ba18fe642787e7b561238e5ba91feb94ad974f13b77cb73f0eff27d WatchSource:0}: Error finding container c0e344b95ba18fe642787e7b561238e5ba91feb94ad974f13b77cb73f0eff27d: Status 404 returned error can't find the container with id c0e344b95ba18fe642787e7b561238e5ba91feb94ad974f13b77cb73f0eff27d Aug 04 00:02:44.254748 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:44.254719 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda966195c_aac5_44da_9599_8bad5ef0e2c3.slice/crio-670927169b7413a9296317b1a73d54115333f5b1c97c897d553dfbb259a1ca4f WatchSource:0}: Error finding container 670927169b7413a9296317b1a73d54115333f5b1c97c897d553dfbb259a1ca4f: Status 404 returned error can't find the container with id 670927169b7413a9296317b1a73d54115333f5b1c97c897d553dfbb259a1ca4f Aug 04 00:02:44.257295 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:44.257206 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc72d8b6_1df2_48fb_b76e_671a7c4b905e.slice/crio-2e5f89f6557191a3aaeea2cace528a4e09ac8c0e2f66ec75eb61aa813773c41f WatchSource:0}: Error finding container 2e5f89f6557191a3aaeea2cace528a4e09ac8c0e2f66ec75eb61aa813773c41f: Status 404 returned error can't find the container with id 2e5f89f6557191a3aaeea2cace528a4e09ac8c0e2f66ec75eb61aa813773c41f Aug 04 00:02:44.261462 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:44.261420 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3404f652_a2a1_46b1_9130_a67db57919f0.slice/crio-bd05fd5284715803cf2a45f08e728b388e4a5cc4b1d66f7de2bdc3cd705ea376 WatchSource:0}: Error finding container bd05fd5284715803cf2a45f08e728b388e4a5cc4b1d66f7de2bdc3cd705ea376: Status 404 returned error can't find the container with id bd05fd5284715803cf2a45f08e728b388e4a5cc4b1d66f7de2bdc3cd705ea376 Aug 04 00:02:44.262399 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:44.262375 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6725222_fca9_49d7_b2c5_9b172b300071.slice/crio-c62bf08f5cf98bc3f4ee74a73cc06cea13bd1e6ce2257110823cf7d9b3438ebe WatchSource:0}: Error finding container c62bf08f5cf98bc3f4ee74a73cc06cea13bd1e6ce2257110823cf7d9b3438ebe: Status 404 returned error can't find the container with id c62bf08f5cf98bc3f4ee74a73cc06cea13bd1e6ce2257110823cf7d9b3438ebe Aug 04 00:02:44.286661 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:44.286637 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2a217d_3a8b_4dea_93ce_b7a478c9f0ef.slice/crio-81f581c81abde83413e9eeffc3f4bb576389d913027b11269c9a810efde96221 WatchSource:0}: Error finding container 81f581c81abde83413e9eeffc3f4bb576389d913027b11269c9a810efde96221: Status 404 returned error can't find the container with id 81f581c81abde83413e9eeffc3f4bb576389d913027b11269c9a810efde96221 Aug 04 00:02:44.287683 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:44.287662 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6920ae2c_7e06_47e6_9101_47f93c1b3226.slice/crio-62343952a3b970db7ad9ade603861e53b4cf10aa46bcccf5e253539a5dbc191d WatchSource:0}: Error finding container 62343952a3b970db7ad9ade603861e53b4cf10aa46bcccf5e253539a5dbc191d: Status 404 returned error can't find the container with id 62343952a3b970db7ad9ade603861e53b4cf10aa46bcccf5e253539a5dbc191d Aug 04 00:02:44.288673 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:44.288649 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4179408d_c308_41a7_b4a1_7446d7cd2a35.slice/crio-81c5a7c2939f14db74d6a5081aa6f0a2fb3c86174e3b4437b44563e5d58c786a WatchSource:0}: Error finding container 81c5a7c2939f14db74d6a5081aa6f0a2fb3c86174e3b4437b44563e5d58c786a: Status 404 returned error can't find the container with id 81c5a7c2939f14db74d6a5081aa6f0a2fb3c86174e3b4437b44563e5d58c786a Aug 04 00:02:44.289565 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:02:44.289540 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a77bedb_69cf_4cd2_8a73_79829ed0c663.slice/crio-d4f791400ea5e12460feaf85e555954c5dc61b82ef3c9203bad7e5dce4d9b35c WatchSource:0}: Error finding container d4f791400ea5e12460feaf85e555954c5dc61b82ef3c9203bad7e5dce4d9b35c: Status 404 returned error can't find the container with id d4f791400ea5e12460feaf85e555954c5dc61b82ef3c9203bad7e5dce4d9b35c Aug 04 00:02:44.381404 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.381366 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:44.381620 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:44.381549 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:44.381620 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:44.381574 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:44.381620 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:44.381588 2552 projected.go:196] Error preparing data for projected volume kube-api-access-gz9jd for pod openshift-network-diagnostics/network-check-target-kwjfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:44.381850 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:44.381656 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd podName:f81f2275-73c2-4b81-8589-ef82bf8cf6ca nodeName:}" failed. No retries permitted until 2026-08-04 00:02:45.381639002 +0000 UTC m=+4.044548464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz9jd" (UniqueName: "kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd") pod "network-check-target-kwjfz" (UID: "f81f2275-73c2-4b81-8589-ef82bf8cf6ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:44.482251 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.482215 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:44.482251 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.482262 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:44.482505 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:44.482379 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:44.482505 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:44.482437 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret podName:c93f255c-fe84-43d8-8ebd-4c54a072e1f8 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:45.482422716 +0000 UTC m=+4.145332173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret") pod "global-pull-secret-syncer-nhd8z" (UID: "c93f255c-fe84-43d8-8ebd-4c54a072e1f8") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:44.482843 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:44.482808 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:44.482991 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:44.482888 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs podName:fc8dfa3c-04cf-4384-ab5f-6cbee7582434 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:45.482868698 +0000 UTC m=+4.145778172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs") pod "network-metrics-daemon-zwpn4" (UID: "fc8dfa3c-04cf-4384-ab5f-6cbee7582434") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:44.814266 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.814164 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-02 23:57:43 +0000 UTC" deadline="2028-03-25 10:37:45.440236443 +0000 UTC" Aug 04 00:02:44.814266 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.814201 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14386h35m0.626040598s" Aug 04 00:02:44.947023 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.946981 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:44.947204 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:44.947122 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:02:44.958552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.958514 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5fnhn" event={"ID":"e2b0ffc0-bd27-4fac-af16-724ba66c64de","Type":"ContainerStarted","Data":"c0e344b95ba18fe642787e7b561238e5ba91feb94ad974f13b77cb73f0eff27d"} Aug 04 00:02:44.961625 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.961558 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mprw6" event={"ID":"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef","Type":"ContainerStarted","Data":"81f581c81abde83413e9eeffc3f4bb576389d913027b11269c9a810efde96221"} Aug 04 00:02:44.964135 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.964046 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" event={"ID":"3404f652-a2a1-46b1-9130-a67db57919f0","Type":"ContainerStarted","Data":"bd05fd5284715803cf2a45f08e728b388e4a5cc4b1d66f7de2bdc3cd705ea376"} Aug 04 00:02:44.969009 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.968939 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4wxgh" event={"ID":"f6725222-fca9-49d7-b2c5-9b172b300071","Type":"ContainerStarted","Data":"c62bf08f5cf98bc3f4ee74a73cc06cea13bd1e6ce2257110823cf7d9b3438ebe"} Aug 04 00:02:44.971675 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.971649 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fkmvm" event={"ID":"cc72d8b6-1df2-48fb-b76e-671a7c4b905e","Type":"ContainerStarted","Data":"2e5f89f6557191a3aaeea2cace528a4e09ac8c0e2f66ec75eb61aa813773c41f"} Aug 04 00:02:44.973848 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.973823 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" event={"ID":"4179408d-c308-41a7-b4a1-7446d7cd2a35","Type":"ContainerStarted","Data":"81c5a7c2939f14db74d6a5081aa6f0a2fb3c86174e3b4437b44563e5d58c786a"} Aug 04 00:02:44.976098 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.976048 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v8btt" event={"ID":"6920ae2c-7e06-47e6-9101-47f93c1b3226","Type":"ContainerStarted","Data":"62343952a3b970db7ad9ade603861e53b4cf10aa46bcccf5e253539a5dbc191d"} Aug 04 00:02:44.977600 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.977574 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vz67" event={"ID":"9a77bedb-69cf-4cd2-8a73-79829ed0c663","Type":"ContainerStarted","Data":"d4f791400ea5e12460feaf85e555954c5dc61b82ef3c9203bad7e5dce4d9b35c"} Aug 04 00:02:44.982775 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:44.982744 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" event={"ID":"a966195c-aac5-44da-9599-8bad5ef0e2c3","Type":"ContainerStarted","Data":"670927169b7413a9296317b1a73d54115333f5b1c97c897d553dfbb259a1ca4f"} Aug 04 00:02:45.391232 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:45.390674 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:45.391232 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.390786 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:45.391232 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.390812 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:45.391232 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.390824 2552 projected.go:196] Error preparing data for projected volume kube-api-access-gz9jd for pod openshift-network-diagnostics/network-check-target-kwjfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:45.391232 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.390984 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd podName:f81f2275-73c2-4b81-8589-ef82bf8cf6ca nodeName:}" failed. No retries permitted until 2026-08-04 00:02:47.390867241 +0000 UTC m=+6.053776712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz9jd" (UniqueName: "kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd") pod "network-check-target-kwjfz" (UID: "f81f2275-73c2-4b81-8589-ef82bf8cf6ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:45.491145 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:45.491117 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:45.491260 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:45.491168 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:45.491327 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.491313 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:45.491392 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.491370 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret podName:c93f255c-fe84-43d8-8ebd-4c54a072e1f8 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:47.491351313 +0000 UTC m=+6.154260785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret") pod "global-pull-secret-syncer-nhd8z" (UID: "c93f255c-fe84-43d8-8ebd-4c54a072e1f8") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:45.491463 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.491430 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:45.491463 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.491462 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs podName:fc8dfa3c-04cf-4384-ab5f-6cbee7582434 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:47.491450137 +0000 UTC m=+6.154359606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs") pod "network-metrics-daemon-zwpn4" (UID: "fc8dfa3c-04cf-4384-ab5f-6cbee7582434") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:45.815059 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:45.815008 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-02 23:57:43 +0000 UTC" deadline="2028-01-03 12:01:48.337336451 +0000 UTC" Aug 04 00:02:45.815059 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:45.815052 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12419h59m2.522288641s" Aug 04 00:02:45.948506 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:45.947982 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:45.948506 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.948166 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:02:45.949012 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:45.948868 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:45.949012 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:45.948968 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:02:45.999095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:45.998424 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"01b58fa67d743aa84130ed84e63b60ea","Type":"ContainerStarted","Data":"43cb4f9fb5454215fdddec33d0dd8b637e0608d98df9b994d8fddf3ea64e96b1"} Aug 04 00:02:46.005239 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:46.004851 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" event={"ID":"7c14b132a47c877cb6e709b2d262ede1","Type":"ContainerStarted","Data":"118d8e385b8f2adecc23244d9757e3d9329059a9d5cded298e636e88f67602a7"} Aug 04 00:02:46.947646 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:46.947611 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:46.948099 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:46.947747 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:02:47.012015 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:47.011960 2552 generic.go:356] "Generic (PLEG): container finished" podID="01b58fa67d743aa84130ed84e63b60ea" containerID="43cb4f9fb5454215fdddec33d0dd8b637e0608d98df9b994d8fddf3ea64e96b1" exitCode=0 Aug 04 00:02:47.013168 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:47.012957 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"01b58fa67d743aa84130ed84e63b60ea","Type":"ContainerDied","Data":"43cb4f9fb5454215fdddec33d0dd8b637e0608d98df9b994d8fddf3ea64e96b1"} Aug 04 00:02:47.034416 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:47.033521 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" podStartSLOduration=4.03350067 podStartE2EDuration="4.03350067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:46.027184579 +0000 UTC m=+4.690094059" watchObservedRunningTime="2026-08-04 00:02:47.03350067 +0000 UTC m=+5.696410151" Aug 04 00:02:47.414998 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:47.414906 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:47.415175 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.415152 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:47.415175 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.415173 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:47.415264 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.415186 2552 projected.go:196] Error preparing data for projected volume kube-api-access-gz9jd for pod openshift-network-diagnostics/network-check-target-kwjfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:47.415264 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.415251 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd podName:f81f2275-73c2-4b81-8589-ef82bf8cf6ca nodeName:}" failed. No retries permitted until 2026-08-04 00:02:51.415230195 +0000 UTC m=+10.078139656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz9jd" (UniqueName: "kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd") pod "network-check-target-kwjfz" (UID: "f81f2275-73c2-4b81-8589-ef82bf8cf6ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:47.516126 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:47.516065 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:47.516126 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:47.516117 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:47.516348 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.516232 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:47.516348 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.516290 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret podName:c93f255c-fe84-43d8-8ebd-4c54a072e1f8 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:51.516272655 +0000 UTC m=+10.179182116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret") pod "global-pull-secret-syncer-nhd8z" (UID: "c93f255c-fe84-43d8-8ebd-4c54a072e1f8") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:47.516348 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.516232 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:47.516479 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.516389 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs podName:fc8dfa3c-04cf-4384-ab5f-6cbee7582434 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:51.516374537 +0000 UTC m=+10.179284007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs") pod "network-metrics-daemon-zwpn4" (UID: "fc8dfa3c-04cf-4384-ab5f-6cbee7582434") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:47.947436 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:47.947398 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:47.947600 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.947556 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:02:47.947808 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:47.947784 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:47.948185 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:47.947899 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:02:48.947693 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:48.947659 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:48.947890 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:48.947796 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:02:49.947932 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:49.947775 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:49.947932 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:49.947846 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:49.948407 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:49.947986 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:02:49.948407 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:49.948094 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:02:50.947068 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:50.947022 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:50.947246 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:50.947170 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:02:51.446306 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:51.446263 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:51.446745 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.446429 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:51.446745 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.446447 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:51.446745 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.446459 2552 projected.go:196] Error preparing data for projected volume kube-api-access-gz9jd for pod openshift-network-diagnostics/network-check-target-kwjfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:51.446745 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.446518 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd podName:f81f2275-73c2-4b81-8589-ef82bf8cf6ca nodeName:}" failed. No retries permitted until 2026-08-04 00:02:59.446497107 +0000 UTC m=+18.109406567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz9jd" (UniqueName: "kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd") pod "network-check-target-kwjfz" (UID: "f81f2275-73c2-4b81-8589-ef82bf8cf6ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:51.547459 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:51.547118 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:51.547459 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:51.547174 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:51.547459 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.547307 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:51.547459 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.547310 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:51.547459 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.547375 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret podName:c93f255c-fe84-43d8-8ebd-4c54a072e1f8 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:59.547357853 +0000 UTC m=+18.210267311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret") pod "global-pull-secret-syncer-nhd8z" (UID: "c93f255c-fe84-43d8-8ebd-4c54a072e1f8") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:51.547459 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.547394 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs podName:fc8dfa3c-04cf-4384-ab5f-6cbee7582434 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:59.547383774 +0000 UTC m=+18.210293237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs") pod "network-metrics-daemon-zwpn4" (UID: "fc8dfa3c-04cf-4384-ab5f-6cbee7582434") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:51.948342 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:51.948318 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:51.948454 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.948430 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:02:51.948592 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:51.948574 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:51.948687 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:51.948654 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:02:52.947720 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:52.947450 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:52.948177 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:52.947788 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:02:53.027056 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.027021 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v8btt" event={"ID":"6920ae2c-7e06-47e6-9101-47f93c1b3226","Type":"ContainerStarted","Data":"9934036478ce7d653f62bfa9ee81d6156c11628f715ce500af18665fec79e02e"} Aug 04 00:02:53.030246 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.030214 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vz67" event={"ID":"9a77bedb-69cf-4cd2-8a73-79829ed0c663","Type":"ContainerStarted","Data":"46dcb9d5f32339b902e726d8d806addd5927acf4d56ad09871ce8d5ae22a01f2"} Aug 04 00:02:53.033235 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.033211 2552 generic.go:356] "Generic (PLEG): container finished" podID="a966195c-aac5-44da-9599-8bad5ef0e2c3" containerID="c676276f1bc9a4265f5b50cadb586bbe5471b91fd4dd168a70384f2898d9234c" exitCode=0 Aug 04 00:02:53.033334 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.033286 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" event={"ID":"a966195c-aac5-44da-9599-8bad5ef0e2c3","Type":"ContainerDied","Data":"c676276f1bc9a4265f5b50cadb586bbe5471b91fd4dd168a70384f2898d9234c"} Aug 04 00:02:53.036669 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.036642 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"01b58fa67d743aa84130ed84e63b60ea","Type":"ContainerStarted","Data":"9f3bdb4bfa94f5d7958d78941683fd21a691813cc150df33676dfad908b437e2"} Aug 04 00:02:53.039349 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.039326 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" event={"ID":"3404f652-a2a1-46b1-9130-a67db57919f0","Type":"ContainerStarted","Data":"7b90b38c38bc357f01aa99c474ca56e8ab13dc78f2e8b678254bb7366a20cd8a"} Aug 04 00:02:53.041192 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.041160 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fkmvm" event={"ID":"cc72d8b6-1df2-48fb-b76e-671a7c4b905e","Type":"ContainerStarted","Data":"7f597b468973c8c05596e15446f0f55ecd5fad9b7010dcc08e57388fe8928f6b"} Aug 04 00:02:53.064211 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.064163 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v8btt" podStartSLOduration=3.4114588169999998 podStartE2EDuration="11.06414608s" totalImagesPullingTime="7.652687263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:53.050272886 +0000 UTC m=+11.713182364" watchObservedRunningTime="2026-08-04 00:02:53.06414608 +0000 UTC m=+11.727055568" Aug 04 00:02:53.064838 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.064804 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8vz67" podStartSLOduration=3.411812512 podStartE2EDuration="11.064796317s" totalImagesPullingTime="7.652983805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:53.063994844 +0000 UTC m=+11.726904324" watchObservedRunningTime="2026-08-04 00:02:53.064796317 +0000 UTC m=+11.727705796" Aug 04 00:02:53.099060 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.099020 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fkmvm" podStartSLOduration=3.41214278 podStartE2EDuration="11.099006819s" totalImagesPullingTime="7.686864039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:53.098624128 +0000 UTC m=+11.761533607" watchObservedRunningTime="2026-08-04 00:02:53.099006819 +0000 UTC m=+11.761916276" Aug 04 00:02:53.113236 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.113191 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" podStartSLOduration=10.113174487 podStartE2EDuration="10.113174487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:53.11276604 +0000 UTC m=+11.775675516" watchObservedRunningTime="2026-08-04 00:02:53.113174487 +0000 UTC m=+11.776083965" Aug 04 00:02:53.947141 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.947075 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:53.947289 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:53.947084 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:53.947289 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:53.947195 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:02:53.947289 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:53.947259 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:02:54.044551 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:54.044511 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5fnhn" event={"ID":"e2b0ffc0-bd27-4fac-af16-724ba66c64de","Type":"ContainerStarted","Data":"ab6036a63a03b9c93756238d266d7fd541de51ead600f57fcaa6d86a174f6ec1"} Aug 04 00:02:54.059844 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:54.059795 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5fnhn" podStartSLOduration=4.362300264 podStartE2EDuration="12.059778974s" totalImagesPullingTime="7.69747871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:54.059671872 +0000 UTC m=+12.722581351" watchObservedRunningTime="2026-08-04 00:02:54.059778974 +0000 UTC m=+12.722688455" Aug 04 00:02:54.587946 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:54.587908 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:54.588553 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:54.588531 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:02:54.946759 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:54.946673 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:54.946930 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:54.946792 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:02:55.946802 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:55.946772 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:55.947437 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:55.946927 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:02:55.947437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:55.947029 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:55.947437 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:55.947128 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:02:56.048650 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:56.048621 2552 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 00:02:56.947415 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:56.947376 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:56.948051 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:56.947507 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:02:57.947392 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:57.947357 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:57.947547 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:57.947408 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:57.947547 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:57.947509 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:02:57.948159 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:57.947649 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:02:58.947103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:58.947069 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:58.947345 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:58.947194 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:02:59.510418 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:59.510382 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:59.510830 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.510527 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:59.510830 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.510548 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:59.510830 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.510560 2552 projected.go:196] Error preparing data for projected volume kube-api-access-gz9jd for pod openshift-network-diagnostics/network-check-target-kwjfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:59.510830 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.510615 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd podName:f81f2275-73c2-4b81-8589-ef82bf8cf6ca nodeName:}" failed. No retries permitted until 2026-08-04 00:03:15.510596614 +0000 UTC m=+34.173506078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz9jd" (UniqueName: "kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd") pod "network-check-target-kwjfz" (UID: "f81f2275-73c2-4b81-8589-ef82bf8cf6ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:59.611399 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:59.611364 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:59.611569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:59.611409 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:02:59.611569 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.611529 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:59.611569 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.611539 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:59.611707 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.611595 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret podName:c93f255c-fe84-43d8-8ebd-4c54a072e1f8 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:15.611581211 +0000 UTC m=+34.274490667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret") pod "global-pull-secret-syncer-nhd8z" (UID: "c93f255c-fe84-43d8-8ebd-4c54a072e1f8") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:59.611707 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.611609 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs podName:fc8dfa3c-04cf-4384-ab5f-6cbee7582434 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:15.61160217 +0000 UTC m=+34.274511625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs") pod "network-metrics-daemon-zwpn4" (UID: "fc8dfa3c-04cf-4384-ab5f-6cbee7582434") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:59.953871 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:59.953778 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:02:59.954032 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.953923 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:02:59.954032 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:02:59.953973 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:02:59.954144 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:02:59.954080 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:00.946956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:00.946927 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:00.947368 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:00.947036 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:03:01.948445 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:01.948414 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:01.948905 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:01.948518 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:01.948905 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:01.948567 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:03:01.948905 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:01.948631 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:02.946875 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:02.946828 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:02.947020 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:02.946946 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:03:03.934591 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:03.934380 2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 00:03:03.946926 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:03.946905 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:03.947048 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:03.946911 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:03.947048 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:03.947031 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:03.947172 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:03.947101 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:03:04.066300 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:04.066254 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" event={"ID":"3404f652-a2a1-46b1-9130-a67db57919f0","Type":"ContainerStarted","Data":"cd2d83ce321b168c34966c8241f2c62641e689bed09c3ddd22c9b1e01f6fb624"} Aug 04 00:03:04.839214 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:04.839092 2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T00:03:03.934540769Z","UUID":"3e82d06a-9d19-4d7c-a820-302c2f7d0498","Handler":null,"Name":"","Endpoint":""} Aug 04 00:03:04.841043 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:04.840996 2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 00:03:04.841227 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:04.841204 2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 00:03:04.854797 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:04.854769 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:03:04.854943 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:04.854930 2552 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 00:03:04.855367 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:04.855343 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8vz67" Aug 04 00:03:04.946868 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:04.946839 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:04.947331 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:04.946978 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:03:05.947707 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:05.947432 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:05.948162 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:05.947482 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:05.948162 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:05.947795 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:05.948162 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:05.947821 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:03:06.071942 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.071907 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" event={"ID":"4179408d-c308-41a7-b4a1-7446d7cd2a35","Type":"ContainerStarted","Data":"23499e67d0fd6401de68a06d4d0658e488ba0461c28ab67d12e479da3563b3fb"} Aug 04 00:03:06.072048 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.072000 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" event={"ID":"4179408d-c308-41a7-b4a1-7446d7cd2a35","Type":"ContainerStarted","Data":"6bef81a89660bd216388a390aff8c6feb4d842e6b476079d2987b23d6820f772"} Aug 04 00:03:06.072048 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.072042 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" event={"ID":"4179408d-c308-41a7-b4a1-7446d7cd2a35","Type":"ContainerStarted","Data":"b286d411646d1a6562ff4466eb7d1614079396dad777d78aa1b5df1d539564fe"} Aug 04 00:03:06.072141 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.072086 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" event={"ID":"4179408d-c308-41a7-b4a1-7446d7cd2a35","Type":"ContainerStarted","Data":"df6fb3fbda88c475f760cc50929d906f64ceafebcb06c87c7568d41f89a3c568"} Aug 04 00:03:06.072141 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.072130 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" event={"ID":"4179408d-c308-41a7-b4a1-7446d7cd2a35","Type":"ContainerStarted","Data":"eeb347109145780ebeb9e807402c03c98f432006d843f6bf37e39c4624d73b13"} Aug 04 00:03:06.073473 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.073430 2552 generic.go:356] "Generic (PLEG): container finished" podID="a966195c-aac5-44da-9599-8bad5ef0e2c3" containerID="670b829f6dfb4654babdb27524b83ccdf68c301b94650da2fc45eacf50c2e04d" exitCode=0 Aug 04 00:03:06.073565 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.073517 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" event={"ID":"a966195c-aac5-44da-9599-8bad5ef0e2c3","Type":"ContainerDied","Data":"670b829f6dfb4654babdb27524b83ccdf68c301b94650da2fc45eacf50c2e04d"} Aug 04 00:03:06.075010 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.074990 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mprw6" event={"ID":"1f2a217d-3a8b-4dea-93ce-b7a478c9f0ef","Type":"ContainerStarted","Data":"0d38adc537b7c9a46217e1dcd7d38d5355ad0115fd4f0ef74a46cd308efb588c"} Aug 04 00:03:06.076303 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.076282 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4wxgh" event={"ID":"f6725222-fca9-49d7-b2c5-9b172b300071","Type":"ContainerStarted","Data":"fb2664d86d338621bd6ca63b2d5d6f04b607ce5435fe45b83a0fb4a2f206579d"} Aug 04 00:03:06.111974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.111928 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4wxgh" podStartSLOduration=4.582582475 podStartE2EDuration="24.111914433s" totalImagesPullingTime="19.529331958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:06.111792699 +0000 UTC m=+24.774702176" watchObservedRunningTime="2026-08-04 00:03:06.111914433 +0000 UTC m=+24.774823911" Aug 04 00:03:06.127276 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.127237 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mprw6" podStartSLOduration=4.589675354 podStartE2EDuration="24.127223841s" totalImagesPullingTime="19.537548487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:06.127045171 +0000 UTC m=+24.789954650" watchObservedRunningTime="2026-08-04 00:03:06.127223841 +0000 UTC m=+24.790133316" Aug 04 00:03:06.946841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:06.946814 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:06.947004 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:06.946919 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:03:07.080418 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:07.080385 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" event={"ID":"4179408d-c308-41a7-b4a1-7446d7cd2a35","Type":"ContainerStarted","Data":"f9b35276c2f29f56e7256f571124e3cc6bdf5d827810cfd076c87e738884f529"} Aug 04 00:03:07.082427 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:07.082399 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" event={"ID":"3404f652-a2a1-46b1-9130-a67db57919f0","Type":"ContainerStarted","Data":"b388e67674ceb90547d843c4611d628f6b7b7cbc6b0a7c47f6cbc42057ac34c2"} Aug 04 00:03:07.100474 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:07.100437 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx2mc" podStartSLOduration=14.067889854 podStartE2EDuration="25.100422698s" totalImagesPullingTime="11.032532844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:07.100017562 +0000 UTC m=+25.762927040" watchObservedRunningTime="2026-08-04 00:03:07.100422698 +0000 UTC m=+25.763332175" Aug 04 00:03:07.946850 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:07.946818 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:07.947042 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:07.946818 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:07.947042 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:07.946954 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:03:07.947118 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:07.947077 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:08.086255 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:08.086218 2552 generic.go:356] "Generic (PLEG): container finished" podID="a966195c-aac5-44da-9599-8bad5ef0e2c3" containerID="bbe7ba1ff985404360e88afc31248f54e529da8755b5cc1ab556819bde552e50" exitCode=0 Aug 04 00:03:08.086816 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:08.086304 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" event={"ID":"a966195c-aac5-44da-9599-8bad5ef0e2c3","Type":"ContainerDied","Data":"bbe7ba1ff985404360e88afc31248f54e529da8755b5cc1ab556819bde552e50"} Aug 04 00:03:08.946952 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:08.946919 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:08.947115 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:08.947055 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:03:09.091223 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:09.091196 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" event={"ID":"4179408d-c308-41a7-b4a1-7446d7cd2a35","Type":"ContainerStarted","Data":"411763dfef3492ceeb6fb1853fce1aaab633f503265b32e8540d9682a533ceee"} Aug 04 00:03:09.947582 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:09.947555 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:09.947791 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:09.947694 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:09.947791 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:09.947740 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:09.947881 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:09.947830 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:03:10.095310 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:10.095278 2552 generic.go:356] "Generic (PLEG): container finished" podID="a966195c-aac5-44da-9599-8bad5ef0e2c3" containerID="cd00e8da457f2eca5611d1b5e6129f51fb2ba4b5057754c7adc243f60cf60039" exitCode=0 Aug 04 00:03:10.095310 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:10.095313 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" event={"ID":"a966195c-aac5-44da-9599-8bad5ef0e2c3","Type":"ContainerDied","Data":"cd00e8da457f2eca5611d1b5e6129f51fb2ba4b5057754c7adc243f60cf60039"} Aug 04 00:03:10.947894 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:10.947644 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:10.948074 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:10.948011 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:03:11.102147 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:11.102118 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" event={"ID":"4179408d-c308-41a7-b4a1-7446d7cd2a35","Type":"ContainerStarted","Data":"1f8b2e21a558cee87084f3dc8ad5d496d4ecb52ed9ff2d3ee1105b3ee5440e41"} Aug 04 00:03:11.102674 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:11.102396 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:03:11.102674 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:11.102488 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:03:11.120151 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:11.120125 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:03:11.129231 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:11.129189 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" podStartSLOduration=9.575435493 podStartE2EDuration="29.129174851s" totalImagesPullingTime="19.553739358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:11.128564929 +0000 UTC m=+29.791474408" watchObservedRunningTime="2026-08-04 00:03:11.129174851 +0000 UTC m=+29.792084328" Aug 04 00:03:11.948952 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:11.948914 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:11.949119 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:11.949050 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:11.949119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:11.949099 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:11.949235 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:11.949204 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:03:12.104824 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:12.104792 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:03:12.122332 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:12.122301 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:03:12.752024 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:12.751991 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kwjfz"] Aug 04 00:03:12.752178 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:12.752157 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:12.752341 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:12.752268 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:03:12.754652 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:12.754629 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zwpn4"] Aug 04 00:03:12.754753 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:12.754739 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:12.754886 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:12.754842 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:12.763803 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:12.763746 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nhd8z"] Aug 04 00:03:12.763917 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:12.763826 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:12.763968 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:12.763926 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:03:14.947808 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:14.947670 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:14.948157 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:14.947720 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:14.948157 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:14.947905 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:03:14.948157 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:14.947728 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:14.948157 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:14.947953 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:14.948157 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:14.948015 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:03:15.114112 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:15.114079 2552 generic.go:356] "Generic (PLEG): container finished" podID="a966195c-aac5-44da-9599-8bad5ef0e2c3" containerID="d9be893a6144f7dc0d2b0294523841711fc7248873df7eda3166da464bd32cb4" exitCode=0 Aug 04 00:03:15.114256 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:15.114137 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" event={"ID":"a966195c-aac5-44da-9599-8bad5ef0e2c3","Type":"ContainerDied","Data":"d9be893a6144f7dc0d2b0294523841711fc7248873df7eda3166da464bd32cb4"} Aug 04 00:03:15.528959 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:15.528876 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:15.529106 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:15.529025 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:03:15.529106 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:15.529048 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:03:15.529106 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:15.529057 2552 projected.go:196] Error preparing data for projected volume kube-api-access-gz9jd for pod openshift-network-diagnostics/network-check-target-kwjfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:03:15.529201 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:15.529107 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd podName:f81f2275-73c2-4b81-8589-ef82bf8cf6ca nodeName:}" failed. No retries permitted until 2026-08-04 00:03:47.529094066 +0000 UTC m=+66.192003521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz9jd" (UniqueName: "kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd") pod "network-check-target-kwjfz" (UID: "f81f2275-73c2-4b81-8589-ef82bf8cf6ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:03:15.629616 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:15.629591 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:15.629779 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:15.629624 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:15.629779 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:15.629732 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:03:15.629779 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:15.629734 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:03:15.629779 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:15.629782 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret podName:c93f255c-fe84-43d8-8ebd-4c54a072e1f8 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:47.629768281 +0000 UTC m=+66.292677738 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret") pod "global-pull-secret-syncer-nhd8z" (UID: "c93f255c-fe84-43d8-8ebd-4c54a072e1f8") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:03:15.629938 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:15.629795 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs podName:fc8dfa3c-04cf-4384-ab5f-6cbee7582434 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:47.62978857 +0000 UTC m=+66.292698026 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs") pod "network-metrics-daemon-zwpn4" (UID: "fc8dfa3c-04cf-4384-ab5f-6cbee7582434") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:03:16.119758 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:16.119720 2552 generic.go:356] "Generic (PLEG): container finished" podID="a966195c-aac5-44da-9599-8bad5ef0e2c3" containerID="58fb07597aeb60eec5d8499403980d3a1078abe1b32c69295c7b5f4fdac948fe" exitCode=0 Aug 04 00:03:16.119758 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:16.119756 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" event={"ID":"a966195c-aac5-44da-9599-8bad5ef0e2c3","Type":"ContainerDied","Data":"58fb07597aeb60eec5d8499403980d3a1078abe1b32c69295c7b5f4fdac948fe"} Aug 04 00:03:16.947624 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:16.947594 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:16.947789 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:16.947594 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:16.947789 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:16.947699 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:03:16.947789 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:16.947594 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:16.947918 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:16.947783 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhd8z" podUID="c93f255c-fe84-43d8-8ebd-4c54a072e1f8" Aug 04 00:03:16.947918 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:16.947834 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwjfz" podUID="f81f2275-73c2-4b81-8589-ef82bf8cf6ca" Aug 04 00:03:17.125074 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:17.125038 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" event={"ID":"a966195c-aac5-44da-9599-8bad5ef0e2c3","Type":"ContainerStarted","Data":"8d2fd5f9e212dfe19ad62dcdcec8864174d19ead9db7235870c7d2e23f213611"} Aug 04 00:03:17.149997 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:17.149893 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cqmnm" podStartSLOduration=8.059229251 podStartE2EDuration="35.149874422s" totalImagesPullingTime="27.090645171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:17.149544131 +0000 UTC m=+35.812453643" watchObservedRunningTime="2026-08-04 00:03:17.149874422 +0000 UTC m=+35.812783892" Aug 04 00:03:18.078955 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.078885 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeReady" Aug 04 00:03:18.079092 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.078990 2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 00:03:18.124827 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.124792 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5bn6k"] Aug 04 00:03:18.142181 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.142150 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wsntm"] Aug 04 00:03:18.142714 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.142235 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.145106 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.145079 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 00:03:18.145869 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.145845 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7ds4t\"" Aug 04 00:03:18.150229 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.149595 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 00:03:18.166209 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.166182 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5bn6k"] Aug 04 00:03:18.166336 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.166283 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wsntm" Aug 04 00:03:18.169675 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.169654 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 00:03:18.169804 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.169710 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 00:03:18.169911 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.169654 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n4xmt\"" Aug 04 00:03:18.170463 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.170446 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 00:03:18.190739 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.190716 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wsntm"] Aug 04 00:03:18.250590 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.250560 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ed5491-dd76-40c3-b091-cd27679ab793-tmp-dir\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.250724 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.250612 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8kw\" (UniqueName: \"kubernetes.io/projected/29ed5491-dd76-40c3-b091-cd27679ab793-kube-api-access-nk8kw\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.250874 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.250846 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.250946 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.250890 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29ed5491-dd76-40c3-b091-cd27679ab793-config-volume\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.351903 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.351811 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjswq\" (UniqueName: \"kubernetes.io/projected/1ed4fcbd-979e-40a5-be0d-11d65f7c4053-kube-api-access-bjswq\") pod \"ingress-canary-wsntm\" (UID: \"1ed4fcbd-979e-40a5-be0d-11d65f7c4053\") " pod="openshift-ingress-canary/ingress-canary-wsntm" Aug 04 00:03:18.351903 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.351888 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ed4fcbd-979e-40a5-be0d-11d65f7c4053-cert\") pod \"ingress-canary-wsntm\" (UID: \"1ed4fcbd-979e-40a5-be0d-11d65f7c4053\") " pod="openshift-ingress-canary/ingress-canary-wsntm" Aug 04 00:03:18.352051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.351932 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ed5491-dd76-40c3-b091-cd27679ab793-tmp-dir\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.352051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.351977 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8kw\" (UniqueName: \"kubernetes.io/projected/29ed5491-dd76-40c3-b091-cd27679ab793-kube-api-access-nk8kw\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.352051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.352015 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.352051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.352036 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29ed5491-dd76-40c3-b091-cd27679ab793-config-volume\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.352233 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:18.352169 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:18.352233 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.352188 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ed5491-dd76-40c3-b091-cd27679ab793-tmp-dir\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.352346 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:18.352246 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls podName:29ed5491-dd76-40c3-b091-cd27679ab793 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:18.85222506 +0000 UTC m=+37.515134522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls") pod "dns-default-5bn6k" (UID: "29ed5491-dd76-40c3-b091-cd27679ab793") : secret "dns-default-metrics-tls" not found Aug 04 00:03:18.352469 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.352446 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29ed5491-dd76-40c3-b091-cd27679ab793-config-volume\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.362067 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.362039 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8kw\" (UniqueName: \"kubernetes.io/projected/29ed5491-dd76-40c3-b091-cd27679ab793-kube-api-access-nk8kw\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.453050 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.453023 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ed4fcbd-979e-40a5-be0d-11d65f7c4053-cert\") pod \"ingress-canary-wsntm\" (UID: \"1ed4fcbd-979e-40a5-be0d-11d65f7c4053\") " pod="openshift-ingress-canary/ingress-canary-wsntm" Aug 04 00:03:18.453187 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.453097 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjswq\" (UniqueName: \"kubernetes.io/projected/1ed4fcbd-979e-40a5-be0d-11d65f7c4053-kube-api-access-bjswq\") pod \"ingress-canary-wsntm\" (UID: \"1ed4fcbd-979e-40a5-be0d-11d65f7c4053\") " pod="openshift-ingress-canary/ingress-canary-wsntm" Aug 04 00:03:18.455277 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.455255 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ed4fcbd-979e-40a5-be0d-11d65f7c4053-cert\") pod \"ingress-canary-wsntm\" (UID: \"1ed4fcbd-979e-40a5-be0d-11d65f7c4053\") " pod="openshift-ingress-canary/ingress-canary-wsntm" Aug 04 00:03:18.462531 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.462509 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjswq\" (UniqueName: \"kubernetes.io/projected/1ed4fcbd-979e-40a5-be0d-11d65f7c4053-kube-api-access-bjswq\") pod \"ingress-canary-wsntm\" (UID: \"1ed4fcbd-979e-40a5-be0d-11d65f7c4053\") " pod="openshift-ingress-canary/ingress-canary-wsntm" Aug 04 00:03:18.474913 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.474891 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wsntm" Aug 04 00:03:18.664747 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.664584 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wsntm"] Aug 04 00:03:18.668707 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:03:18.668682 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed4fcbd_979e_40a5_be0d_11d65f7c4053.slice/crio-bfd531ad8dbc5abe14ff0aa3ee20bb58ee0afe37c654c5f48294d7ac2818a064 WatchSource:0}: Error finding container bfd531ad8dbc5abe14ff0aa3ee20bb58ee0afe37c654c5f48294d7ac2818a064: Status 404 returned error can't find the container with id bfd531ad8dbc5abe14ff0aa3ee20bb58ee0afe37c654c5f48294d7ac2818a064 Aug 04 00:03:18.856592 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.856557 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:18.856731 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:18.856660 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:18.856731 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:18.856710 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls podName:29ed5491-dd76-40c3-b091-cd27679ab793 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:19.856696313 +0000 UTC m=+38.519605770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls") pod "dns-default-5bn6k" (UID: "29ed5491-dd76-40c3-b091-cd27679ab793") : secret "dns-default-metrics-tls" not found Aug 04 00:03:18.947360 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.947282 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:18.947360 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.947326 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:18.947516 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.947282 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:18.950054 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.950031 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 00:03:18.950173 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.950031 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 00:03:18.950173 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.950062 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 00:03:18.950308 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.950294 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 00:03:18.950374 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.950309 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9tct\"" Aug 04 00:03:18.950426 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:18.950396 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qjt6l\"" Aug 04 00:03:19.131034 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:19.130999 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wsntm" event={"ID":"1ed4fcbd-979e-40a5-be0d-11d65f7c4053","Type":"ContainerStarted","Data":"bfd531ad8dbc5abe14ff0aa3ee20bb58ee0afe37c654c5f48294d7ac2818a064"} Aug 04 00:03:19.869649 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:19.869614 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:19.870146 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:19.869769 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:19.870146 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:19.869849 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls podName:29ed5491-dd76-40c3-b091-cd27679ab793 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:21.869833658 +0000 UTC m=+40.532743118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls") pod "dns-default-5bn6k" (UID: "29ed5491-dd76-40c3-b091-cd27679ab793") : secret "dns-default-metrics-tls" not found Aug 04 00:03:21.881674 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:21.881500 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:21.882046 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:21.881627 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:21.882046 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:21.881745 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls podName:29ed5491-dd76-40c3-b091-cd27679ab793 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:25.881728396 +0000 UTC m=+44.544637852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls") pod "dns-default-5bn6k" (UID: "29ed5491-dd76-40c3-b091-cd27679ab793") : secret "dns-default-metrics-tls" not found Aug 04 00:03:22.139478 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:22.139394 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wsntm" event={"ID":"1ed4fcbd-979e-40a5-be0d-11d65f7c4053","Type":"ContainerStarted","Data":"26501f62f2b0c3934ec20a716f856fe7642f59735c2e2a7ccca1378ef16d469c"} Aug 04 00:03:25.908439 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:25.908403 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:25.908926 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:25.908511 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:25.908926 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:25.908561 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls podName:29ed5491-dd76-40c3-b091-cd27679ab793 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:33.90854713 +0000 UTC m=+52.571456586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls") pod "dns-default-5bn6k" (UID: "29ed5491-dd76-40c3-b091-cd27679ab793") : secret "dns-default-metrics-tls" not found Aug 04 00:03:33.966428 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:33.966393 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:33.966903 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:33.966529 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:33.966903 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:33.966586 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls podName:29ed5491-dd76-40c3-b091-cd27679ab793 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:49.966571834 +0000 UTC m=+68.629481290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls") pod "dns-default-5bn6k" (UID: "29ed5491-dd76-40c3-b091-cd27679ab793") : secret "dns-default-metrics-tls" not found Aug 04 00:03:44.119565 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:44.119538 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7tkg" Aug 04 00:03:44.149818 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:44.149771 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wsntm" podStartSLOduration=23.699294034 podStartE2EDuration="26.149755163s" totalImagesPullingTime="2.450461129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:22.153297495 +0000 UTC m=+40.816206972" watchObservedRunningTime="2026-08-04 00:03:44.149755163 +0000 UTC m=+62.812664641" Aug 04 00:03:47.564000 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.563960 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:47.566883 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.566848 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 00:03:47.577087 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.577065 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 00:03:47.588127 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.588103 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9jd\" (UniqueName: \"kubernetes.io/projected/f81f2275-73c2-4b81-8589-ef82bf8cf6ca-kube-api-access-gz9jd\") pod \"network-check-target-kwjfz\" (UID: \"f81f2275-73c2-4b81-8589-ef82bf8cf6ca\") " pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:47.665262 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.665234 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:03:47.665403 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.665273 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:47.667921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.667904 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 00:03:47.668085 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.668070 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 00:03:47.675623 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:47.675607 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 00:03:47.675687 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:47.675676 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs podName:fc8dfa3c-04cf-4384-ab5f-6cbee7582434 nodeName:}" failed. No retries permitted until 2026-08-04 00:04:51.6756572 +0000 UTC m=+130.338566656 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs") pod "network-metrics-daemon-zwpn4" (UID: "fc8dfa3c-04cf-4384-ab5f-6cbee7582434") : secret "metrics-daemon-secret" not found Aug 04 00:03:47.677707 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.677684 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c93f255c-fe84-43d8-8ebd-4c54a072e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-nhd8z\" (UID: \"c93f255c-fe84-43d8-8ebd-4c54a072e1f8\") " pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:47.779276 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.779247 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhd8z" Aug 04 00:03:47.785998 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.785980 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qjt6l\"" Aug 04 00:03:47.794069 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.794052 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:47.959767 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.959703 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nhd8z"] Aug 04 00:03:47.964127 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:03:47.964102 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93f255c_fe84_43d8_8ebd_4c54a072e1f8.slice/crio-8516c7a3540c8fb24707462b339c53f029e625ca7e55300e856edc9464077aca WatchSource:0}: Error finding container 8516c7a3540c8fb24707462b339c53f029e625ca7e55300e856edc9464077aca: Status 404 returned error can't find the container with id 8516c7a3540c8fb24707462b339c53f029e625ca7e55300e856edc9464077aca Aug 04 00:03:47.974451 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:47.974428 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kwjfz"] Aug 04 00:03:47.976976 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:03:47.976960 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81f2275_73c2_4b81_8589_ef82bf8cf6ca.slice/crio-7cf4930744051931e744183108829454fda79dae4610778d9c67347c21154c11 WatchSource:0}: Error finding container 7cf4930744051931e744183108829454fda79dae4610778d9c67347c21154c11: Status 404 returned error can't find the container with id 7cf4930744051931e744183108829454fda79dae4610778d9c67347c21154c11 Aug 04 00:03:48.201224 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:48.201193 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kwjfz" event={"ID":"f81f2275-73c2-4b81-8589-ef82bf8cf6ca","Type":"ContainerStarted","Data":"7cf4930744051931e744183108829454fda79dae4610778d9c67347c21154c11"} Aug 04 00:03:48.202165 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:48.202140 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nhd8z" event={"ID":"c93f255c-fe84-43d8-8ebd-4c54a072e1f8","Type":"ContainerStarted","Data":"8516c7a3540c8fb24707462b339c53f029e625ca7e55300e856edc9464077aca"} Aug 04 00:03:49.981604 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:49.981574 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:03:49.982058 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:49.981768 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:49.982058 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:03:49.981829 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls podName:29ed5491-dd76-40c3-b091-cd27679ab793 nodeName:}" failed. No retries permitted until 2026-08-04 00:04:21.981810492 +0000 UTC m=+100.644719953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls") pod "dns-default-5bn6k" (UID: "29ed5491-dd76-40c3-b091-cd27679ab793") : secret "dns-default-metrics-tls" not found Aug 04 00:03:53.216652 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:53.216615 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nhd8z" event={"ID":"c93f255c-fe84-43d8-8ebd-4c54a072e1f8","Type":"ContainerStarted","Data":"7df2a6df52c542ee011e5058f8c8846bf168045c649193b616605da72ea19590"} Aug 04 00:03:53.218051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:53.218026 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kwjfz" event={"ID":"f81f2275-73c2-4b81-8589-ef82bf8cf6ca","Type":"ContainerStarted","Data":"59248dee74996894a1d9c52af348137630f996f8267191f9926e3d9b8743156c"} Aug 04 00:03:53.218165 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:53.218081 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:03:53.232369 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:53.232327 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nhd8z" podStartSLOduration=66.496992744 podStartE2EDuration="1m11.232313995s" totalImagesPullingTime="4.735321251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:53.231525642 +0000 UTC m=+71.894435130" watchObservedRunningTime="2026-08-04 00:03:53.232313995 +0000 UTC m=+71.895223472" Aug 04 00:03:53.245377 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:03:53.245338 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kwjfz" podStartSLOduration=66.526308265 podStartE2EDuration="1m11.24532513s" totalImagesPullingTime="4.719016865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:53.245081964 +0000 UTC m=+71.907991445" watchObservedRunningTime="2026-08-04 00:03:53.24532513 +0000 UTC m=+71.908234585" Aug 04 00:04:03.501451 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.501320 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2"] Aug 04 00:04:03.504180 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.504158 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght"] Aug 04 00:04:03.504310 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.504280 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.506887 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.506869 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.506974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.506894 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 00:04:03.506974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.506908 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 00:04:03.506974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.506918 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 00:04:03.507147 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.507036 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Aug 04 00:04:03.509303 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.509284 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 04 00:04:03.509905 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.509886 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 04 00:04:03.509987 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.509925 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 04 00:04:03.509987 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.509961 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 04 00:04:03.512985 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.512964 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2"] Aug 04 00:04:03.516013 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.515995 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght"] Aug 04 00:04:03.572836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.572811 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8knr2\" (UniqueName: \"kubernetes.io/projected/950378e9-9faa-4c0f-ad1d-113003485c30-kube-api-access-8knr2\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.572983 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.572843 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-hub\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.572983 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.572883 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.572983 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.572959 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.573087 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.573021 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-ca\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.573087 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.573047 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/feca2f0e-d777-456b-ba48-cc408860f09e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f8f44c9f9-ds4v2\" (UID: \"feca2f0e-d777-456b-ba48-cc408860f09e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.573087 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.573078 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feca2f0e-d777-456b-ba48-cc408860f09e-tmp\") pod \"klusterlet-addon-workmgr-7f8f44c9f9-ds4v2\" (UID: \"feca2f0e-d777-456b-ba48-cc408860f09e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.573198 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.573094 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2nn\" (UniqueName: \"kubernetes.io/projected/feca2f0e-d777-456b-ba48-cc408860f09e-kube-api-access-7g2nn\") pod \"klusterlet-addon-workmgr-7f8f44c9f9-ds4v2\" (UID: \"feca2f0e-d777-456b-ba48-cc408860f09e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.573198 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.573155 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/950378e9-9faa-4c0f-ad1d-113003485c30-additional-service-ca\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.674301 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674279 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-ca\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.674410 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674306 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/feca2f0e-d777-456b-ba48-cc408860f09e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f8f44c9f9-ds4v2\" (UID: \"feca2f0e-d777-456b-ba48-cc408860f09e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.674410 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674331 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feca2f0e-d777-456b-ba48-cc408860f09e-tmp\") pod \"klusterlet-addon-workmgr-7f8f44c9f9-ds4v2\" (UID: \"feca2f0e-d777-456b-ba48-cc408860f09e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.674410 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674346 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2nn\" (UniqueName: \"kubernetes.io/projected/feca2f0e-d777-456b-ba48-cc408860f09e-kube-api-access-7g2nn\") pod \"klusterlet-addon-workmgr-7f8f44c9f9-ds4v2\" (UID: \"feca2f0e-d777-456b-ba48-cc408860f09e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.674520 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674481 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/950378e9-9faa-4c0f-ad1d-113003485c30-additional-service-ca\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.674559 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674533 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8knr2\" (UniqueName: \"kubernetes.io/projected/950378e9-9faa-4c0f-ad1d-113003485c30-kube-api-access-8knr2\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.674613 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674567 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-hub\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.674705 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674685 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.674768 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674709 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feca2f0e-d777-456b-ba48-cc408860f09e-tmp\") pod \"klusterlet-addon-workmgr-7f8f44c9f9-ds4v2\" (UID: \"feca2f0e-d777-456b-ba48-cc408860f09e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.674820 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.674776 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.675247 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.675226 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/950378e9-9faa-4c0f-ad1d-113003485c30-additional-service-ca\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.677053 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.677023 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-hub\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.677143 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.677122 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.677212 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.677191 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-ca\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.677358 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.677338 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/feca2f0e-d777-456b-ba48-cc408860f09e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f8f44c9f9-ds4v2\" (UID: \"feca2f0e-d777-456b-ba48-cc408860f09e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.677478 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.677466 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/950378e9-9faa-4c0f-ad1d-113003485c30-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.682184 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.682162 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8knr2\" (UniqueName: \"kubernetes.io/projected/950378e9-9faa-4c0f-ad1d-113003485c30-kube-api-access-8knr2\") pod \"cluster-proxy-proxy-agent-777fd56967-skght\" (UID: \"950378e9-9faa-4c0f-ad1d-113003485c30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.682307 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.682291 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2nn\" (UniqueName: \"kubernetes.io/projected/feca2f0e-d777-456b-ba48-cc408860f09e-kube-api-access-7g2nn\") pod \"klusterlet-addon-workmgr-7f8f44c9f9-ds4v2\" (UID: \"feca2f0e-d777-456b-ba48-cc408860f09e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.815592 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.815535 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:03.829580 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.829557 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:04:03.950946 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.950821 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2"] Aug 04 00:04:03.953001 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:04:03.952980 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeca2f0e_d777_456b_ba48_cc408860f09e.slice/crio-e57aea431a3940a04113d62f4f5a439bfbd7e09a0e3b37905ffa1bc94081876a WatchSource:0}: Error finding container e57aea431a3940a04113d62f4f5a439bfbd7e09a0e3b37905ffa1bc94081876a: Status 404 returned error can't find the container with id e57aea431a3940a04113d62f4f5a439bfbd7e09a0e3b37905ffa1bc94081876a Aug 04 00:04:03.962139 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:03.962118 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght"] Aug 04 00:04:03.965003 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:04:03.964982 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod950378e9_9faa_4c0f_ad1d_113003485c30.slice/crio-22ccce962a55e4644fe522ac0e47af3bdf42e252f0002aab5a77d149d8148d55 WatchSource:0}: Error finding container 22ccce962a55e4644fe522ac0e47af3bdf42e252f0002aab5a77d149d8148d55: Status 404 returned error can't find the container with id 22ccce962a55e4644fe522ac0e47af3bdf42e252f0002aab5a77d149d8148d55 Aug 04 00:04:04.248530 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:04.248498 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" event={"ID":"950378e9-9faa-4c0f-ad1d-113003485c30","Type":"ContainerStarted","Data":"22ccce962a55e4644fe522ac0e47af3bdf42e252f0002aab5a77d149d8148d55"} Aug 04 00:04:04.249495 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:04.249464 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" event={"ID":"feca2f0e-d777-456b-ba48-cc408860f09e","Type":"ContainerStarted","Data":"e57aea431a3940a04113d62f4f5a439bfbd7e09a0e3b37905ffa1bc94081876a"} Aug 04 00:04:10.277106 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:10.277004 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" event={"ID":"950378e9-9faa-4c0f-ad1d-113003485c30","Type":"ContainerStarted","Data":"6435c78c7bd25eb55f9ee0a3e17aaf7d0737e6e8cf412badd8555884ab1aea3b"} Aug 04 00:04:10.277106 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:10.277052 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" event={"ID":"950378e9-9faa-4c0f-ad1d-113003485c30","Type":"ContainerStarted","Data":"dec915ee4298ef79a8bf0f92a1d3712dc735a635654b2fbce8d414a0720c0680"} Aug 04 00:04:10.277631 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:10.277117 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" event={"ID":"950378e9-9faa-4c0f-ad1d-113003485c30","Type":"ContainerStarted","Data":"7114998b3147e20229e15ef1d6e9dfa634fdb1219473e53097548760aea1fd74"} Aug 04 00:04:10.278603 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:10.278573 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" event={"ID":"feca2f0e-d777-456b-ba48-cc408860f09e","Type":"ContainerStarted","Data":"9985ae886fd7f190d36165c7d29bee0bb106dc9b0265125339fb9df54bd2e690"} Aug 04 00:04:10.278881 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:10.278661 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:10.280037 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:10.280018 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:04:10.313402 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:10.313361 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" podStartSLOduration=1.198021705 podStartE2EDuration="7.313348829s" totalImagesPullingTime="6.115327124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:04:10.312738107 +0000 UTC m=+88.975647584" watchObservedRunningTime="2026-08-04 00:04:10.313348829 +0000 UTC m=+88.976258307" Aug 04 00:04:10.313616 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:10.313597 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" podStartSLOduration=1.209561316 podStartE2EDuration="7.313593723s" totalImagesPullingTime="6.104032407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:04:10.298535382 +0000 UTC m=+88.961444860" watchObservedRunningTime="2026-08-04 00:04:10.313593723 +0000 UTC m=+88.976503198" Aug 04 00:04:21.999382 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:21.999340 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:04:22.000019 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:04:21.999440 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:04:22.000019 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:04:21.999502 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls podName:29ed5491-dd76-40c3-b091-cd27679ab793 nodeName:}" failed. No retries permitted until 2026-08-04 00:05:25.999483237 +0000 UTC m=+164.662392693 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls") pod "dns-default-5bn6k" (UID: "29ed5491-dd76-40c3-b091-cd27679ab793") : secret "dns-default-metrics-tls" not found Aug 04 00:04:24.222645 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:24.222619 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kwjfz" Aug 04 00:04:51.702825 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:04:51.702787 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:04:51.703315 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:04:51.702964 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 00:04:51.703315 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:04:51.703047 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs podName:fc8dfa3c-04cf-4384-ab5f-6cbee7582434 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:53.703025282 +0000 UTC m=+252.365934742 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs") pod "network-metrics-daemon-zwpn4" (UID: "fc8dfa3c-04cf-4384-ab5f-6cbee7582434") : secret "metrics-daemon-secret" not found Aug 04 00:05:21.153135 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:05:21.153089 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5bn6k" podUID="29ed5491-dd76-40c3-b091-cd27679ab793" Aug 04 00:05:21.477665 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:05:21.477591 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bn6k" Aug 04 00:05:21.968403 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:05:21.968366 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zwpn4" podUID="fc8dfa3c-04cf-4384-ab5f-6cbee7582434" Aug 04 00:05:26.037271 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:05:26.037238 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:05:26.037658 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:05:26.037388 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:05:26.037658 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:05:26.037463 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls podName:29ed5491-dd76-40c3-b091-cd27679ab793 nodeName:}" failed. No retries permitted until 2026-08-04 00:07:28.037446483 +0000 UTC m=+286.700355942 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls") pod "dns-default-5bn6k" (UID: "29ed5491-dd76-40c3-b091-cd27679ab793") : secret "dns-default-metrics-tls" not found Aug 04 00:05:33.947467 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:05:33.947384 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:06:10.279080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:10.279027 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" podUID="feca2f0e-d777-456b-ba48-cc408860f09e" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused" Aug 04 00:06:10.613762 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:10.613733 2552 generic.go:356] "Generic (PLEG): container finished" podID="feca2f0e-d777-456b-ba48-cc408860f09e" containerID="9985ae886fd7f190d36165c7d29bee0bb106dc9b0265125339fb9df54bd2e690" exitCode=1 Aug 04 00:06:10.613932 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:10.613770 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" event={"ID":"feca2f0e-d777-456b-ba48-cc408860f09e","Type":"ContainerDied","Data":"9985ae886fd7f190d36165c7d29bee0bb106dc9b0265125339fb9df54bd2e690"} Aug 04 00:06:10.614120 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:10.614108 2552 scope.go:117] "RemoveContainer" containerID="9985ae886fd7f190d36165c7d29bee0bb106dc9b0265125339fb9df54bd2e690" Aug 04 00:06:11.619216 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:11.619184 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" event={"ID":"feca2f0e-d777-456b-ba48-cc408860f09e","Type":"ContainerStarted","Data":"255a0ce2509e166b68a47166dce4ae7ff7e2775c9cb010c6a31b3932300c1d50"} Aug 04 00:06:11.619618 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:11.619550 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:06:11.620098 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:11.620079 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8f44c9f9-ds4v2" Aug 04 00:06:39.153975 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.153943 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kdbcj"] Aug 04 00:06:39.155949 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.155927 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.158353 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.158334 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 00:06:39.158439 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.158334 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 00:06:39.159341 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.159319 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 00:06:39.159341 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.159340 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbnpj\"" Aug 04 00:06:39.159493 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.159328 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 00:06:39.167387 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.167360 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kdbcj"] Aug 04 00:06:39.242928 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.242900 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2545bce7-2c88-4073-9cd9-733ad69b540c-data-volume\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.243066 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.242945 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2545bce7-2c88-4073-9cd9-733ad69b540c-crio-socket\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.243066 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.242963 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jf75\" (UniqueName: \"kubernetes.io/projected/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-api-access-2jf75\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.243066 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.243031 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.243066 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.243046 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.343748 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.343712 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2545bce7-2c88-4073-9cd9-733ad69b540c-crio-socket\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.343748 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.343751 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jf75\" (UniqueName: \"kubernetes.io/projected/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-api-access-2jf75\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.343956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.343817 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.343956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.343841 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.343956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.343843 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2545bce7-2c88-4073-9cd9-733ad69b540c-crio-socket\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.343956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.343897 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2545bce7-2c88-4073-9cd9-733ad69b540c-data-volume\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.343956 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:39.343934 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:39.344135 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:39.343984 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.344135 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:39.344014 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:39.843995978 +0000 UTC m=+238.506905450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : configmap "kube-rbac-proxy" not found Aug 04 00:06:39.344135 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:39.344042 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:39.844028605 +0000 UTC m=+238.506938065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.344245 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.344229 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2545bce7-2c88-4073-9cd9-733ad69b540c-data-volume\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.354009 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.353992 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jf75\" (UniqueName: \"kubernetes.io/projected/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-api-access-2jf75\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.847740 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.847692 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.847740 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:39.847740 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:39.847979 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:39.847819 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:39.847979 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:39.847845 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.847979 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:39.847916 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:40.847897945 +0000 UTC m=+239.510807419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.847979 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:39.847934 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:40.847925719 +0000 UTC m=+239.510835175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : configmap "kube-rbac-proxy" not found Aug 04 00:06:40.854305 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:40.854268 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:40.854305 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:40.854305 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:40.854739 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:40.854389 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:40.854739 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:40.854452 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:40.854739 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:40.854457 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.85443611 +0000 UTC m=+241.517345577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : configmap "kube-rbac-proxy" not found Aug 04 00:06:40.854739 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:40.854495 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.854485889 +0000 UTC m=+241.517395347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:42.118623 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:42.118596 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v8btt_6920ae2c-7e06-47e6-9101-47f93c1b3226/dns-node-resolver/0.log" Aug 04 00:06:42.870942 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:42.870907 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:42.870942 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:42.870942 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:42.871151 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:42.871034 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:42.871151 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:42.871061 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:42.871151 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:42.871110 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:46.871083358 +0000 UTC m=+245.533992824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : configmap "kube-rbac-proxy" not found Aug 04 00:06:42.871151 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:42.871124 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:46.871117952 +0000 UTC m=+245.534027408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:43.119435 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:43.119411 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fkmvm_cc72d8b6-1df2-48fb-b76e-671a7c4b905e/node-ca/0.log" Aug 04 00:06:44.119216 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:44.119185 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wsntm_1ed4fcbd-979e-40a5-be0d-11d65f7c4053/serve-healthcheck-canary/0.log" Aug 04 00:06:46.902092 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:46.902061 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:46.902092 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:46.902096 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:46.902601 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:46.902176 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:46.902601 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:46.902187 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:46.902601 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:46.902249 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:54.902231992 +0000 UTC m=+253.565141447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:46.902601 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:06:46.902272 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm podName:2545bce7-2c88-4073-9cd9-733ad69b540c nodeName:}" failed. No retries permitted until 2026-08-04 00:06:54.902258953 +0000 UTC m=+253.565168409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kdbcj" (UID: "2545bce7-2c88-4073-9cd9-733ad69b540c") : configmap "kube-rbac-proxy" not found Aug 04 00:06:53.751229 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:53.751198 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:06:53.753425 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:53.753406 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8dfa3c-04cf-4384-ab5f-6cbee7582434-metrics-certs\") pod \"network-metrics-daemon-zwpn4\" (UID: \"fc8dfa3c-04cf-4384-ab5f-6cbee7582434\") " pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:06:54.050574 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:54.050485 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9tct\"" Aug 04 00:06:54.058748 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:54.058727 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwpn4" Aug 04 00:06:54.175762 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:54.175733 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zwpn4"] Aug 04 00:06:54.178771 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:06:54.178743 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8dfa3c_04cf_4384_ab5f_6cbee7582434.slice/crio-63cf83a9b9071ab21ab5139ab9b0d127d92bb93f9f68d6565bce13e816040e82 WatchSource:0}: Error finding container 63cf83a9b9071ab21ab5139ab9b0d127d92bb93f9f68d6565bce13e816040e82: Status 404 returned error can't find the container with id 63cf83a9b9071ab21ab5139ab9b0d127d92bb93f9f68d6565bce13e816040e82 Aug 04 00:06:54.739226 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:54.739194 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwpn4" event={"ID":"fc8dfa3c-04cf-4384-ab5f-6cbee7582434","Type":"ContainerStarted","Data":"63cf83a9b9071ab21ab5139ab9b0d127d92bb93f9f68d6565bce13e816040e82"} Aug 04 00:06:54.961590 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:54.961548 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:54.962036 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:54.961600 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:54.962245 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:54.962219 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2545bce7-2c88-4073-9cd9-733ad69b540c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:54.964406 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:54.964376 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2545bce7-2c88-4073-9cd9-733ad69b540c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdbcj\" (UID: \"2545bce7-2c88-4073-9cd9-733ad69b540c\") " pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:55.066903 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:55.066816 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbnpj\"" Aug 04 00:06:55.074666 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:55.074646 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kdbcj" Aug 04 00:06:55.214276 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:55.214235 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kdbcj"] Aug 04 00:06:55.218533 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:06:55.218502 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2545bce7_2c88_4073_9cd9_733ad69b540c.slice/crio-5112c23f669f1074013b62bfcdecf6c4c377c1ce06beff6c6c011aced0e2b725 WatchSource:0}: Error finding container 5112c23f669f1074013b62bfcdecf6c4c377c1ce06beff6c6c011aced0e2b725: Status 404 returned error can't find the container with id 5112c23f669f1074013b62bfcdecf6c4c377c1ce06beff6c6c011aced0e2b725 Aug 04 00:06:55.743939 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:55.743903 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kdbcj" event={"ID":"2545bce7-2c88-4073-9cd9-733ad69b540c","Type":"ContainerStarted","Data":"cdfa5cd93186681506617a1246167799895a9ecb031b3a4bbe09361b81811aa2"} Aug 04 00:06:55.744114 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:55.743970 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kdbcj" event={"ID":"2545bce7-2c88-4073-9cd9-733ad69b540c","Type":"ContainerStarted","Data":"5112c23f669f1074013b62bfcdecf6c4c377c1ce06beff6c6c011aced0e2b725"} Aug 04 00:06:55.747449 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:55.747417 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwpn4" event={"ID":"fc8dfa3c-04cf-4384-ab5f-6cbee7582434","Type":"ContainerStarted","Data":"1c15702cd36035166597a2e82c189b0aa35c433b0384b7bdc2943da9b6b4f8f6"} Aug 04 00:06:55.747602 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:55.747473 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwpn4" event={"ID":"fc8dfa3c-04cf-4384-ab5f-6cbee7582434","Type":"ContainerStarted","Data":"4e05a9c0611fddb13173b2f30a5e3233f6cb7063b2a51738111c0a163382d6a9"} Aug 04 00:06:55.764949 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:55.764908 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zwpn4" podStartSLOduration=252.86194684 podStartE2EDuration="4m13.7648511s" totalImagesPullingTime="902.90426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:55.763671591 +0000 UTC m=+254.426581069" watchObservedRunningTime="2026-08-04 00:06:55.7648511 +0000 UTC m=+254.427760578" Aug 04 00:06:56.753278 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:56.753241 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kdbcj" event={"ID":"2545bce7-2c88-4073-9cd9-733ad69b540c","Type":"ContainerStarted","Data":"eecc0ef93395dd9be3cb18e3379af27da4fe83834f37472c4e8714f0e3d934d1"} Aug 04 00:06:57.757664 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:57.757629 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kdbcj" event={"ID":"2545bce7-2c88-4073-9cd9-733ad69b540c","Type":"ContainerStarted","Data":"15bf52e1a0984cc009fecf27ee4e37c07524aeb7a962afc0f341f6f666348980"} Aug 04 00:06:57.775113 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:06:57.775059 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kdbcj" podStartSLOduration=16.773186568 podStartE2EDuration="18.775042614s" totalImagesPullingTime="2.001856046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:06:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:57.774094712 +0000 UTC m=+256.437004191" watchObservedRunningTime="2026-08-04 00:06:57.775042614 +0000 UTC m=+256.437952091" Aug 04 00:07:03.460269 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.460188 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75ff4b669d-kbgg5"] Aug 04 00:07:03.462712 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.462691 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.465369 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.465348 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 00:07:03.465470 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.465393 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rbxcz\"" Aug 04 00:07:03.465470 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.465436 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 00:07:03.466447 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.466419 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 00:07:03.471136 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.471117 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 00:07:03.475298 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.475279 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75ff4b669d-kbgg5"] Aug 04 00:07:03.521371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.521342 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-registry-certificates\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.521371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.521375 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-trusted-ca\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.521533 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.521396 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-image-registry-private-configuration\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.521533 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.521429 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-bound-sa-token\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.521533 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.521482 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-registry-tls\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.521533 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.521521 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-ca-trust-extracted\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.521658 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.521539 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-installation-pull-secrets\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.521658 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.521578 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rnxg\" (UniqueName: \"kubernetes.io/projected/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-kube-api-access-2rnxg\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.622389 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.622359 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-registry-certificates\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.622498 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.622396 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-trusted-ca\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.622498 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.622423 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-image-registry-private-configuration\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.622498 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.622450 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-bound-sa-token\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.622633 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.622501 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-registry-tls\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.622633 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.622549 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-ca-trust-extracted\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.622633 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.622570 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-installation-pull-secrets\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.622633 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.622592 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rnxg\" (UniqueName: \"kubernetes.io/projected/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-kube-api-access-2rnxg\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.623130 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.623094 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-ca-trust-extracted\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.625096 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.623610 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-registry-certificates\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.625096 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.623719 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-trusted-ca\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.625722 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.625693 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-registry-tls\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.625833 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.625695 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-installation-pull-secrets\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.626077 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.626056 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-image-registry-private-configuration\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.635574 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.635553 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-bound-sa-token\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.636169 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.636148 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rnxg\" (UniqueName: \"kubernetes.io/projected/1ac3d4fe-04d7-46bf-96f8-81dec40df02f-kube-api-access-2rnxg\") pod \"image-registry-75ff4b669d-kbgg5\" (UID: \"1ac3d4fe-04d7-46bf-96f8-81dec40df02f\") " pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.773041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.772969 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:03.905653 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:03.905633 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75ff4b669d-kbgg5"] Aug 04 00:07:03.908076 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:07:03.908051 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac3d4fe_04d7_46bf_96f8_81dec40df02f.slice/crio-391124eed646b47df9a82cbd8da17be6bffb4f8ecd15d386a7eca329706ddfdf WatchSource:0}: Error finding container 391124eed646b47df9a82cbd8da17be6bffb4f8ecd15d386a7eca329706ddfdf: Status 404 returned error can't find the container with id 391124eed646b47df9a82cbd8da17be6bffb4f8ecd15d386a7eca329706ddfdf Aug 04 00:07:04.779742 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:04.779707 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" event={"ID":"1ac3d4fe-04d7-46bf-96f8-81dec40df02f","Type":"ContainerStarted","Data":"6a5927ffa682b4f0e1a8a0c90eba99df6b3e9bdbb01a89809ce0b4f8ad65b10a"} Aug 04 00:07:04.780142 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:04.779755 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" event={"ID":"1ac3d4fe-04d7-46bf-96f8-81dec40df02f","Type":"ContainerStarted","Data":"391124eed646b47df9a82cbd8da17be6bffb4f8ecd15d386a7eca329706ddfdf"} Aug 04 00:07:04.780142 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:04.779781 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:04.799631 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:04.799593 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" podStartSLOduration=1.7995782070000002 podStartE2EDuration="1.799578207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:04.797963343 +0000 UTC m=+263.460872820" watchObservedRunningTime="2026-08-04 00:07:04.799578207 +0000 UTC m=+263.462487684" Aug 04 00:07:20.601473 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.601445 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx"] Aug 04 00:07:20.606444 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.606425 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.609278 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.609258 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 00:07:20.609734 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.609717 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 04 00:07:20.609830 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.609751 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 04 00:07:20.610391 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.610373 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pg9ks\"" Aug 04 00:07:20.610709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.610694 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 04 00:07:20.610785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.610736 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 00:07:20.610785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.610750 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 00:07:20.617487 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.617466 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qvgp4"] Aug 04 00:07:20.620718 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.620698 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx"] Aug 04 00:07:20.620812 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.620783 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.623114 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.623089 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 00:07:20.623231 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.623102 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 00:07:20.623298 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.623241 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 00:07:20.623601 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.623589 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6k5hv\"" Aug 04 00:07:20.638574 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.638553 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pkw\" (UniqueName: \"kubernetes.io/projected/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-api-access-w8pkw\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.638661 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.638595 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/90e765d3-ddca-4bf1-867e-b1468bd64b63-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.638701 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.638673 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.638736 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.638705 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.638769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.638738 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90e765d3-ddca-4bf1-867e-b1468bd64b63-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.638769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.638759 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.739513 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739488 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.739638 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739522 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.739638 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739545 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.739638 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739569 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.739638 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739591 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90e765d3-ddca-4bf1-867e-b1468bd64b63-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.739638 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739613 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45955895-bd01-4566-8ace-f5ccdc90e24a-root\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.739638 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739628 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45955895-bd01-4566-8ace-f5ccdc90e24a-metrics-client-ca\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.739921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739651 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.739921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739695 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqg2\" (UniqueName: \"kubernetes.io/projected/45955895-bd01-4566-8ace-f5ccdc90e24a-kube-api-access-4nqg2\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.739921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739726 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-wtmp\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.739921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739757 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pkw\" (UniqueName: \"kubernetes.io/projected/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-api-access-w8pkw\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.739921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739802 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45955895-bd01-4566-8ace-f5ccdc90e24a-sys\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.739921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739871 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-textfile\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.739921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739903 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-tls\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.740242 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.739974 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/90e765d3-ddca-4bf1-867e-b1468bd64b63-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.740302 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.740286 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/90e765d3-ddca-4bf1-867e-b1468bd64b63-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.740363 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.740336 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.740513 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.740495 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90e765d3-ddca-4bf1-867e-b1468bd64b63-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.742444 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.742419 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.742552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.742531 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.748736 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.748715 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pkw\" (UniqueName: \"kubernetes.io/projected/90e765d3-ddca-4bf1-867e-b1468bd64b63-kube-api-access-w8pkw\") pod \"kube-state-metrics-6bd9664c4-j7krx\" (UID: \"90e765d3-ddca-4bf1-867e-b1468bd64b63\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:20.840941 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.840915 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-tls\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841078 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.840975 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841078 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:07:20.841071 2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 00:07:20.841170 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:07:20.841131 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-tls podName:45955895-bd01-4566-8ace-f5ccdc90e24a nodeName:}" failed. No retries permitted until 2026-08-04 00:07:21.341116566 +0000 UTC m=+280.004026021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-tls") pod "node-exporter-qvgp4" (UID: "45955895-bd01-4566-8ace-f5ccdc90e24a") : secret "node-exporter-tls" not found Aug 04 00:07:20.841170 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841151 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841252 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841198 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45955895-bd01-4566-8ace-f5ccdc90e24a-root\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841252 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841223 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45955895-bd01-4566-8ace-f5ccdc90e24a-metrics-client-ca\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841332 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841259 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqg2\" (UniqueName: \"kubernetes.io/projected/45955895-bd01-4566-8ace-f5ccdc90e24a-kube-api-access-4nqg2\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841332 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841272 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45955895-bd01-4566-8ace-f5ccdc90e24a-root\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841332 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841287 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-wtmp\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841332 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841328 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45955895-bd01-4566-8ace-f5ccdc90e24a-sys\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841353 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-textfile\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841418 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-wtmp\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841429 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45955895-bd01-4566-8ace-f5ccdc90e24a-sys\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841674 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841636 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-textfile\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841764 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841746 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.841842 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.841820 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45955895-bd01-4566-8ace-f5ccdc90e24a-metrics-client-ca\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.843115 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.843092 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.852745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.852702 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqg2\" (UniqueName: \"kubernetes.io/projected/45955895-bd01-4566-8ace-f5ccdc90e24a-kube-api-access-4nqg2\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:20.915783 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:20.915758 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" Aug 04 00:07:21.042458 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:21.042422 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx"] Aug 04 00:07:21.045441 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:07:21.045414 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e765d3_ddca_4bf1_867e_b1468bd64b63.slice/crio-1f9c0f4018bb50a54a236f3b57a7432dc584070a95a32ccb8ec06f89fb4f5ea8 WatchSource:0}: Error finding container 1f9c0f4018bb50a54a236f3b57a7432dc584070a95a32ccb8ec06f89fb4f5ea8: Status 404 returned error can't find the container with id 1f9c0f4018bb50a54a236f3b57a7432dc584070a95a32ccb8ec06f89fb4f5ea8 Aug 04 00:07:21.344687 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:21.344659 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-tls\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:21.346917 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:21.346889 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45955895-bd01-4566-8ace-f5ccdc90e24a-node-exporter-tls\") pod \"node-exporter-qvgp4\" (UID: \"45955895-bd01-4566-8ace-f5ccdc90e24a\") " pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:21.529148 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:21.529112 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qvgp4" Aug 04 00:07:21.540201 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:07:21.540168 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45955895_bd01_4566_8ace_f5ccdc90e24a.slice/crio-ea3605a688fe6529c3a76fa5e287b5517dc10b60d6f2ab0d14ab76c41617b82e WatchSource:0}: Error finding container ea3605a688fe6529c3a76fa5e287b5517dc10b60d6f2ab0d14ab76c41617b82e: Status 404 returned error can't find the container with id ea3605a688fe6529c3a76fa5e287b5517dc10b60d6f2ab0d14ab76c41617b82e Aug 04 00:07:21.833133 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:21.833097 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qvgp4" event={"ID":"45955895-bd01-4566-8ace-f5ccdc90e24a","Type":"ContainerStarted","Data":"ea3605a688fe6529c3a76fa5e287b5517dc10b60d6f2ab0d14ab76c41617b82e"} Aug 04 00:07:21.834086 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:21.834063 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" event={"ID":"90e765d3-ddca-4bf1-867e-b1468bd64b63","Type":"ContainerStarted","Data":"1f9c0f4018bb50a54a236f3b57a7432dc584070a95a32ccb8ec06f89fb4f5ea8"} Aug 04 00:07:22.839427 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:22.839331 2552 generic.go:356] "Generic (PLEG): container finished" podID="45955895-bd01-4566-8ace-f5ccdc90e24a" containerID="3cfdeccb74e605d8c6a0348c96b7edbd487674c550af79b3fdb38f553963fe72" exitCode=0 Aug 04 00:07:22.839848 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:22.839418 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qvgp4" event={"ID":"45955895-bd01-4566-8ace-f5ccdc90e24a","Type":"ContainerDied","Data":"3cfdeccb74e605d8c6a0348c96b7edbd487674c550af79b3fdb38f553963fe72"} Aug 04 00:07:22.841779 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:22.841748 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" event={"ID":"90e765d3-ddca-4bf1-867e-b1468bd64b63","Type":"ContainerStarted","Data":"142c0819f6a6e3f074d5c3f6ebb0cb294e1f5dd6bb53b4ea58d68bfc1ca10db1"} Aug 04 00:07:22.841919 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:22.841808 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" event={"ID":"90e765d3-ddca-4bf1-867e-b1468bd64b63","Type":"ContainerStarted","Data":"dcb79649469f1347482b86d8ed915b44d8d0f9057159818ca433a5a9d8d6f020"} Aug 04 00:07:22.841919 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:22.841842 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" event={"ID":"90e765d3-ddca-4bf1-867e-b1468bd64b63","Type":"ContainerStarted","Data":"aff72b012a10ef33ce7af20ad01f5d3808845f5f5f3ec61231d01abff265a412"} Aug 04 00:07:22.886336 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:22.886097 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-j7krx" podStartSLOduration=1.665233215 podStartE2EDuration="2.886078433s" totalImagesPullingTime="1.220845218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:22.885749512 +0000 UTC m=+281.548658991" watchObservedRunningTime="2026-08-04 00:07:22.886078433 +0000 UTC m=+281.548987914" Aug 04 00:07:23.777118 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:23.777087 2552 patch_prober.go:28] interesting pod/image-registry-75ff4b669d-kbgg5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 00:07:23.777305 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:23.777139 2552 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" podUID="1ac3d4fe-04d7-46bf-96f8-81dec40df02f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:07:23.847241 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:23.847207 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qvgp4" event={"ID":"45955895-bd01-4566-8ace-f5ccdc90e24a","Type":"ContainerStarted","Data":"ef1deb0c23686f954a867ea07073a97b75d8bfb7e111af16c9fb550a7fe4fd54"} Aug 04 00:07:23.847623 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:23.847268 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qvgp4" event={"ID":"45955895-bd01-4566-8ace-f5ccdc90e24a","Type":"ContainerStarted","Data":"0a9e45633fe395a242321a9bca82541e63129eb18aa6ad9132b2ad33f55adf27"} Aug 04 00:07:23.867693 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:23.867646 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qvgp4" podStartSLOduration=2.800711955 podStartE2EDuration="3.867631956s" totalImagesPullingTime="976.717889ms" totalInitContainerRuntime="90.202112ms" isStatefulPod=true podCreationTimestamp="2026-08-04 00:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:23.866073676 +0000 UTC m=+282.528983154" watchObservedRunningTime="2026-08-04 00:07:23.867631956 +0000 UTC m=+282.530541433" Aug 04 00:07:24.478737 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:07:24.478701 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5bn6k" podUID="29ed5491-dd76-40c3-b091-cd27679ab793" Aug 04 00:07:24.850641 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:24.850558 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bn6k" Aug 04 00:07:25.414195 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.414150 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c"] Aug 04 00:07:25.417505 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.417489 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:25.421149 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.421127 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 04 00:07:25.421244 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.421127 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n2f2m\"" Aug 04 00:07:25.430625 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.430604 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c"] Aug 04 00:07:25.478097 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.478062 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/febed18d-372c-4b67-8396-bff767963966-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-2vl2c\" (UID: \"febed18d-372c-4b67-8396-bff767963966\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:25.478243 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.478105 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxhf\" (UniqueName: \"kubernetes.io/projected/febed18d-372c-4b67-8396-bff767963966-kube-api-access-6nxhf\") pod \"monitoring-plugin-6dffbbd67-2vl2c\" (UID: \"febed18d-372c-4b67-8396-bff767963966\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:25.580812 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.580772 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/febed18d-372c-4b67-8396-bff767963966-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-2vl2c\" (UID: \"febed18d-372c-4b67-8396-bff767963966\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:25.580984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.580835 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxhf\" (UniqueName: \"kubernetes.io/projected/febed18d-372c-4b67-8396-bff767963966-kube-api-access-6nxhf\") pod \"monitoring-plugin-6dffbbd67-2vl2c\" (UID: \"febed18d-372c-4b67-8396-bff767963966\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:25.581457 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:07:25.581241 2552 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Aug 04 00:07:25.581457 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:07:25.581313 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/febed18d-372c-4b67-8396-bff767963966-monitoring-plugin-cert podName:febed18d-372c-4b67-8396-bff767963966 nodeName:}" failed. No retries permitted until 2026-08-04 00:07:26.081293417 +0000 UTC m=+284.744202874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/febed18d-372c-4b67-8396-bff767963966-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-2vl2c" (UID: "febed18d-372c-4b67-8396-bff767963966") : secret "monitoring-plugin-cert" not found Aug 04 00:07:25.590342 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.590313 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxhf\" (UniqueName: \"kubernetes.io/projected/febed18d-372c-4b67-8396-bff767963966-kube-api-access-6nxhf\") pod \"monitoring-plugin-6dffbbd67-2vl2c\" (UID: \"febed18d-372c-4b67-8396-bff767963966\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:25.788384 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.788292 2552 patch_prober.go:28] interesting pod/image-registry-75ff4b669d-kbgg5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 00:07:25.788384 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:25.788352 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" podUID="1ac3d4fe-04d7-46bf-96f8-81dec40df02f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:07:26.084432 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:26.084344 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/febed18d-372c-4b67-8396-bff767963966-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-2vl2c\" (UID: \"febed18d-372c-4b67-8396-bff767963966\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:26.086694 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:26.086671 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/febed18d-372c-4b67-8396-bff767963966-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-2vl2c\" (UID: \"febed18d-372c-4b67-8396-bff767963966\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:26.325821 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:26.325786 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:26.447228 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:26.447204 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c"] Aug 04 00:07:26.449736 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:07:26.449712 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebed18d_372c_4b67_8396_bff767963966.slice/crio-808af2886a3b971fe98902102ea486959ab548f33d3a02cf18dbf8366ef27b6c WatchSource:0}: Error finding container 808af2886a3b971fe98902102ea486959ab548f33d3a02cf18dbf8366ef27b6c: Status 404 returned error can't find the container with id 808af2886a3b971fe98902102ea486959ab548f33d3a02cf18dbf8366ef27b6c Aug 04 00:07:26.857627 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:26.857593 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" event={"ID":"febed18d-372c-4b67-8396-bff767963966","Type":"ContainerStarted","Data":"808af2886a3b971fe98902102ea486959ab548f33d3a02cf18dbf8366ef27b6c"} Aug 04 00:07:27.862577 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:27.862544 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" event={"ID":"febed18d-372c-4b67-8396-bff767963966","Type":"ContainerStarted","Data":"381ee6ab0a495437f48e7c67c38ea5f7c3f27f83eb42dcc28a0f1d92e603eecd"} Aug 04 00:07:27.862972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:27.862825 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:27.867546 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:27.867526 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" Aug 04 00:07:27.880804 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:27.880730 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2vl2c" podStartSLOduration=1.703290169 podStartE2EDuration="2.880716556s" totalImagesPullingTime="1.177426387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:27.880476655 +0000 UTC m=+286.543386135" watchObservedRunningTime="2026-08-04 00:07:27.880716556 +0000 UTC m=+286.543626035" Aug 04 00:07:28.100230 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:28.100195 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:07:28.102461 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:28.102433 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ed5491-dd76-40c3-b091-cd27679ab793-metrics-tls\") pod \"dns-default-5bn6k\" (UID: \"29ed5491-dd76-40c3-b091-cd27679ab793\") " pod="openshift-dns/dns-default-5bn6k" Aug 04 00:07:28.154098 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:28.154028 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7ds4t\"" Aug 04 00:07:28.162078 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:28.162061 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bn6k" Aug 04 00:07:28.286641 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:28.286606 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5bn6k"] Aug 04 00:07:28.291248 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:07:28.291216 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ed5491_dd76_40c3_b091_cd27679ab793.slice/crio-84394c3c14960dbe0aa3e5ca68a5f9e9d95d4669e037de8e7085813be8b02584 WatchSource:0}: Error finding container 84394c3c14960dbe0aa3e5ca68a5f9e9d95d4669e037de8e7085813be8b02584: Status 404 returned error can't find the container with id 84394c3c14960dbe0aa3e5ca68a5f9e9d95d4669e037de8e7085813be8b02584 Aug 04 00:07:28.867554 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:28.867521 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bn6k" event={"ID":"29ed5491-dd76-40c3-b091-cd27679ab793","Type":"ContainerStarted","Data":"84394c3c14960dbe0aa3e5ca68a5f9e9d95d4669e037de8e7085813be8b02584"} Aug 04 00:07:30.875304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:30.875269 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bn6k" event={"ID":"29ed5491-dd76-40c3-b091-cd27679ab793","Type":"ContainerStarted","Data":"8787e9509dc6667d45d2f7c03ce6184b650bbb4c867de945209b7e2bef22038a"} Aug 04 00:07:30.875712 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:30.875349 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bn6k" event={"ID":"29ed5491-dd76-40c3-b091-cd27679ab793","Type":"ContainerStarted","Data":"b9019e93ef6ea7a8e3ed304f1cbe591f96a7b8b17ae7999f096c56eed18576ff"} Aug 04 00:07:30.875712 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:30.875414 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5bn6k" Aug 04 00:07:30.893643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:30.893593 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5bn6k" podStartSLOduration=251.146416443 podStartE2EDuration="4m12.89357067s" totalImagesPullingTime="1.747154227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:30.893377905 +0000 UTC m=+289.556287387" watchObservedRunningTime="2026-08-04 00:07:30.89357067 +0000 UTC m=+289.556480149" Aug 04 00:07:33.777652 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:33.777617 2552 patch_prober.go:28] interesting pod/image-registry-75ff4b669d-kbgg5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 00:07:33.778040 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:33.777669 2552 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" podUID="1ac3d4fe-04d7-46bf-96f8-81dec40df02f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:07:35.787397 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:35.787369 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75ff4b669d-kbgg5" Aug 04 00:07:40.881200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:40.881165 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5bn6k" Aug 04 00:07:41.779375 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:41.779348 2552 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 00:07:43.830894 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:43.830831 2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" podUID="950378e9-9faa-4c0f-ad1d-113003485c30" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Aug 04 00:07:49.551251 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:49.551216 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bn6k_29ed5491-dd76-40c3-b091-cd27679ab793/dns/0.log" Aug 04 00:07:49.750313 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:49.750274 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bn6k_29ed5491-dd76-40c3-b091-cd27679ab793/kube-rbac-proxy/0.log" Aug 04 00:07:51.149674 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:51.149643 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v8btt_6920ae2c-7e06-47e6-9101-47f93c1b3226/dns-node-resolver/0.log" Aug 04 00:07:51.951478 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:51.951451 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wsntm_1ed4fcbd-979e-40a5-be0d-11d65f7c4053/serve-healthcheck-canary/0.log" Aug 04 00:07:53.830739 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:07:53.830699 2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" podUID="950378e9-9faa-4c0f-ad1d-113003485c30" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Aug 04 00:08:03.831029 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:08:03.830988 2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" podUID="950378e9-9faa-4c0f-ad1d-113003485c30" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Aug 04 00:08:03.831563 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:08:03.831052 2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" Aug 04 00:08:03.831563 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:08:03.831486 2552 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6435c78c7bd25eb55f9ee0a3e17aaf7d0737e6e8cf412badd8555884ab1aea3b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" containerMessage="Container service-proxy failed liveness probe, will be restarted" Aug 04 00:08:03.831563 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:08:03.831536 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" podUID="950378e9-9faa-4c0f-ad1d-113003485c30" containerName="service-proxy" containerID="cri-o://6435c78c7bd25eb55f9ee0a3e17aaf7d0737e6e8cf412badd8555884ab1aea3b" gracePeriod=30 Aug 04 00:08:03.989552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:08:03.989528 2552 generic.go:356] "Generic (PLEG): container finished" podID="950378e9-9faa-4c0f-ad1d-113003485c30" containerID="6435c78c7bd25eb55f9ee0a3e17aaf7d0737e6e8cf412badd8555884ab1aea3b" exitCode=2 Aug 04 00:08:03.989648 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:08:03.989587 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" event={"ID":"950378e9-9faa-4c0f-ad1d-113003485c30","Type":"ContainerDied","Data":"6435c78c7bd25eb55f9ee0a3e17aaf7d0737e6e8cf412badd8555884ab1aea3b"} Aug 04 00:08:04.996198 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:08:04.996162 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777fd56967-skght" event={"ID":"950378e9-9faa-4c0f-ad1d-113003485c30","Type":"ContainerStarted","Data":"2dfbe5f23569f3782c67db8195c5cb34294244074c87c8e0e927976ace44de7e"} Aug 04 00:10:53.392117 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.392087 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4"] Aug 04 00:10:53.394947 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.394931 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" Aug 04 00:10:53.397656 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.397631 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 00:10:53.397776 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.397734 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-z8s2j\"" Aug 04 00:10:53.398957 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.398920 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:10:53.403082 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.403057 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4"] Aug 04 00:10:53.449508 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.449481 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78wh\" (UniqueName: \"kubernetes.io/projected/1132b218-176c-49ee-9694-eb122741113a-kube-api-access-r78wh\") pod \"openshift-lws-operator-fd8ccff4c-tp5f4\" (UID: \"1132b218-176c-49ee-9694-eb122741113a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" Aug 04 00:10:53.449659 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.449516 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1132b218-176c-49ee-9694-eb122741113a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tp5f4\" (UID: \"1132b218-176c-49ee-9694-eb122741113a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" Aug 04 00:10:53.550053 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.550012 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78wh\" (UniqueName: \"kubernetes.io/projected/1132b218-176c-49ee-9694-eb122741113a-kube-api-access-r78wh\") pod \"openshift-lws-operator-fd8ccff4c-tp5f4\" (UID: \"1132b218-176c-49ee-9694-eb122741113a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" Aug 04 00:10:53.550212 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.550060 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1132b218-176c-49ee-9694-eb122741113a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tp5f4\" (UID: \"1132b218-176c-49ee-9694-eb122741113a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" Aug 04 00:10:53.550432 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.550411 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1132b218-176c-49ee-9694-eb122741113a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tp5f4\" (UID: \"1132b218-176c-49ee-9694-eb122741113a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" Aug 04 00:10:53.558033 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.558009 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78wh\" (UniqueName: \"kubernetes.io/projected/1132b218-176c-49ee-9694-eb122741113a-kube-api-access-r78wh\") pod \"openshift-lws-operator-fd8ccff4c-tp5f4\" (UID: \"1132b218-176c-49ee-9694-eb122741113a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" Aug 04 00:10:53.704180 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.704115 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" Aug 04 00:10:53.829945 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.829923 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4"] Aug 04 00:10:53.832031 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:10:53.832006 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1132b218_176c_49ee_9694_eb122741113a.slice/crio-f3e14d7395d01f9fa0ed853018c2ba6f4cf0727a31363a430205c1ee2c4bb2b9 WatchSource:0}: Error finding container f3e14d7395d01f9fa0ed853018c2ba6f4cf0727a31363a430205c1ee2c4bb2b9: Status 404 returned error can't find the container with id f3e14d7395d01f9fa0ed853018c2ba6f4cf0727a31363a430205c1ee2c4bb2b9 Aug 04 00:10:53.833433 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:53.833414 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:10:54.550002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:54.549972 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" event={"ID":"1132b218-176c-49ee-9694-eb122741113a","Type":"ContainerStarted","Data":"f3e14d7395d01f9fa0ed853018c2ba6f4cf0727a31363a430205c1ee2c4bb2b9"} Aug 04 00:10:59.569599 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:59.569508 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" event={"ID":"1132b218-176c-49ee-9694-eb122741113a","Type":"ContainerStarted","Data":"a63852914b27c263e96cd0aaccf1ae023a5f618bf45ceba087d279185f842883"} Aug 04 00:10:59.587719 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:10:59.587664 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tp5f4" podStartSLOduration=1.230833484 podStartE2EDuration="6.587644013s" totalImagesPullingTime="5.356810529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:10:59.586088001 +0000 UTC m=+498.248997478" watchObservedRunningTime="2026-08-04 00:10:59.587644013 +0000 UTC m=+498.250553493" Aug 04 00:11:17.539739 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.539705 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn"] Aug 04 00:11:17.547184 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.547161 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.550049 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.550025 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 00:11:17.550164 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.550097 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h29qs\"" Aug 04 00:11:17.550321 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.550304 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 00:11:17.550390 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.550372 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 00:11:17.550441 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.550413 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 00:11:17.559597 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.558126 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn"] Aug 04 00:11:17.633966 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.633932 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9efde704-d281-4e4a-8164-160af9c7033a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.634125 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.633978 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9efde704-d281-4e4a-8164-160af9c7033a-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.634125 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.633999 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkx7h\" (UniqueName: \"kubernetes.io/projected/9efde704-d281-4e4a-8164-160af9c7033a-kube-api-access-qkx7h\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.634125 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.634052 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9efde704-d281-4e4a-8164-160af9c7033a-tmp\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.634125 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.634086 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9efde704-d281-4e4a-8164-160af9c7033a-manifests\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.734810 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.734776 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9efde704-d281-4e4a-8164-160af9c7033a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.735018 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.734818 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9efde704-d281-4e4a-8164-160af9c7033a-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.735018 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.734922 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkx7h\" (UniqueName: \"kubernetes.io/projected/9efde704-d281-4e4a-8164-160af9c7033a-kube-api-access-qkx7h\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.735018 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.734989 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9efde704-d281-4e4a-8164-160af9c7033a-tmp\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.735204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.735035 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9efde704-d281-4e4a-8164-160af9c7033a-manifests\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.735416 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.735396 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9efde704-d281-4e4a-8164-160af9c7033a-manifests\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.737249 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.737227 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9efde704-d281-4e4a-8164-160af9c7033a-tmp\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.737366 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.737348 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9efde704-d281-4e4a-8164-160af9c7033a-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.737366 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.737356 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9efde704-d281-4e4a-8164-160af9c7033a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.746154 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.746127 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkx7h\" (UniqueName: \"kubernetes.io/projected/9efde704-d281-4e4a-8164-160af9c7033a-kube-api-access-qkx7h\") pod \"opendatahub-operator-controller-manager-67b98d597c-wgcxn\" (UID: \"9efde704-d281-4e4a-8164-160af9c7033a\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.858955 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.858881 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:17.990221 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:17.990192 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn"] Aug 04 00:11:17.993403 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:11:17.993371 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efde704_d281_4e4a_8164_160af9c7033a.slice/crio-4657c7c0f62ba68d3c2891bde77359bb7d924cfd9d14531129aa43bdac2cf809 WatchSource:0}: Error finding container 4657c7c0f62ba68d3c2891bde77359bb7d924cfd9d14531129aa43bdac2cf809: Status 404 returned error can't find the container with id 4657c7c0f62ba68d3c2891bde77359bb7d924cfd9d14531129aa43bdac2cf809 Aug 04 00:11:18.637110 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:18.637069 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" event={"ID":"9efde704-d281-4e4a-8164-160af9c7033a","Type":"ContainerStarted","Data":"4657c7c0f62ba68d3c2891bde77359bb7d924cfd9d14531129aa43bdac2cf809"} Aug 04 00:11:21.649887 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:21.649836 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" event={"ID":"9efde704-d281-4e4a-8164-160af9c7033a","Type":"ContainerStarted","Data":"28f1781519fbe09aa4af113388e5056b0d520b7b9db39a00bbdaa277e7933d97"} Aug 04 00:11:22.654461 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:22.654429 2552 generic.go:356] "Generic (PLEG): container finished" podID="9efde704-d281-4e4a-8164-160af9c7033a" containerID="28f1781519fbe09aa4af113388e5056b0d520b7b9db39a00bbdaa277e7933d97" exitCode=0 Aug 04 00:11:22.654838 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:22.654504 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" event={"ID":"9efde704-d281-4e4a-8164-160af9c7033a","Type":"ContainerDied","Data":"28f1781519fbe09aa4af113388e5056b0d520b7b9db39a00bbdaa277e7933d97"} Aug 04 00:11:23.659734 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:23.659696 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" event={"ID":"9efde704-d281-4e4a-8164-160af9c7033a","Type":"ContainerStarted","Data":"32a9f80356ec61ffc8987cdb51578da8f644443b1b77cde8bcbfe5f736e88de2"} Aug 04 00:11:23.660156 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:23.659812 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:23.682198 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:23.682156 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" podStartSLOduration=2.452612623 podStartE2EDuration="6.682143919s" totalImagesPullingTime="3.194707658s" totalInitContainerRuntime="1.034823638s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:23.681626986 +0000 UTC m=+522.344536464" watchObservedRunningTime="2026-08-04 00:11:23.682143919 +0000 UTC m=+522.345053397" Aug 04 00:11:34.665465 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:34.665389 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-wgcxn" Aug 04 00:11:36.932070 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:36.932040 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc"] Aug 04 00:11:36.934794 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:36.934779 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:36.937950 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:36.937932 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 00:11:36.938239 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:36.938224 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fdf5h\"" Aug 04 00:11:36.938288 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:36.938253 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 00:11:36.938520 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:36.938506 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 00:11:36.938964 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:36.938951 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 00:11:36.948568 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:36.948546 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc"] Aug 04 00:11:37.011165 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.011139 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gxlrp"] Aug 04 00:11:37.014060 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.014045 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:37.016546 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.016522 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 04 00:11:37.016670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.016525 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9gtmw\"" Aug 04 00:11:37.024849 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.024826 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gxlrp"] Aug 04 00:11:37.081492 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.081461 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20a59f4-5d4e-4796-b7d8-60b9587e8b3d-tls-certs\") pod \"kube-auth-proxy-554b4579cb-f2ksc\" (UID: \"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.081651 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.081504 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f20a59f4-5d4e-4796-b7d8-60b9587e8b3d-tmp\") pod \"kube-auth-proxy-554b4579cb-f2ksc\" (UID: \"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.081651 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.081585 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvs67\" (UniqueName: \"kubernetes.io/projected/f20a59f4-5d4e-4796-b7d8-60b9587e8b3d-kube-api-access-rvs67\") pod \"kube-auth-proxy-554b4579cb-f2ksc\" (UID: \"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.183018 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.182919 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08461dce-ee65-4059-9985-ca2af7252bcb-cert\") pod \"odh-model-controller-7998c67dc4-gxlrp\" (UID: \"08461dce-ee65-4059-9985-ca2af7252bcb\") " pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:37.183018 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.182981 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs67\" (UniqueName: \"kubernetes.io/projected/f20a59f4-5d4e-4796-b7d8-60b9587e8b3d-kube-api-access-rvs67\") pod \"kube-auth-proxy-554b4579cb-f2ksc\" (UID: \"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.183218 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.183101 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20a59f4-5d4e-4796-b7d8-60b9587e8b3d-tls-certs\") pod \"kube-auth-proxy-554b4579cb-f2ksc\" (UID: \"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.183218 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.183133 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f20a59f4-5d4e-4796-b7d8-60b9587e8b3d-tmp\") pod \"kube-auth-proxy-554b4579cb-f2ksc\" (UID: \"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.183218 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.183162 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nldt\" (UniqueName: \"kubernetes.io/projected/08461dce-ee65-4059-9985-ca2af7252bcb-kube-api-access-4nldt\") pod \"odh-model-controller-7998c67dc4-gxlrp\" (UID: \"08461dce-ee65-4059-9985-ca2af7252bcb\") " pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:37.185549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.185526 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f20a59f4-5d4e-4796-b7d8-60b9587e8b3d-tmp\") pod \"kube-auth-proxy-554b4579cb-f2ksc\" (UID: \"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.185762 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.185744 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20a59f4-5d4e-4796-b7d8-60b9587e8b3d-tls-certs\") pod \"kube-auth-proxy-554b4579cb-f2ksc\" (UID: \"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.194811 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.194789 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvs67\" (UniqueName: \"kubernetes.io/projected/f20a59f4-5d4e-4796-b7d8-60b9587e8b3d-kube-api-access-rvs67\") pod \"kube-auth-proxy-554b4579cb-f2ksc\" (UID: \"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.248764 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.248736 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" Aug 04 00:11:37.284171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.284142 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nldt\" (UniqueName: \"kubernetes.io/projected/08461dce-ee65-4059-9985-ca2af7252bcb-kube-api-access-4nldt\") pod \"odh-model-controller-7998c67dc4-gxlrp\" (UID: \"08461dce-ee65-4059-9985-ca2af7252bcb\") " pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:37.284171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.284176 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08461dce-ee65-4059-9985-ca2af7252bcb-cert\") pod \"odh-model-controller-7998c67dc4-gxlrp\" (UID: \"08461dce-ee65-4059-9985-ca2af7252bcb\") " pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:37.284331 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:11:37.284273 2552 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Aug 04 00:11:37.284366 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:11:37.284333 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08461dce-ee65-4059-9985-ca2af7252bcb-cert podName:08461dce-ee65-4059-9985-ca2af7252bcb nodeName:}" failed. No retries permitted until 2026-08-04 00:11:37.784318141 +0000 UTC m=+536.447227597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/08461dce-ee65-4059-9985-ca2af7252bcb-cert") pod "odh-model-controller-7998c67dc4-gxlrp" (UID: "08461dce-ee65-4059-9985-ca2af7252bcb") : secret "odh-model-controller-webhook-cert" not found Aug 04 00:11:37.293665 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.293644 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nldt\" (UniqueName: \"kubernetes.io/projected/08461dce-ee65-4059-9985-ca2af7252bcb-kube-api-access-4nldt\") pod \"odh-model-controller-7998c67dc4-gxlrp\" (UID: \"08461dce-ee65-4059-9985-ca2af7252bcb\") " pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:37.369560 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.369471 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc"] Aug 04 00:11:37.371968 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:11:37.371945 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20a59f4_5d4e_4796_b7d8_60b9587e8b3d.slice/crio-e2209c4705744c987b88cf199858bd40017ff8a63f2a8d43f5b4f7f07b042876 WatchSource:0}: Error finding container e2209c4705744c987b88cf199858bd40017ff8a63f2a8d43f5b4f7f07b042876: Status 404 returned error can't find the container with id e2209c4705744c987b88cf199858bd40017ff8a63f2a8d43f5b4f7f07b042876 Aug 04 00:11:37.709828 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.709789 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" event={"ID":"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d","Type":"ContainerStarted","Data":"e2209c4705744c987b88cf199858bd40017ff8a63f2a8d43f5b4f7f07b042876"} Aug 04 00:11:37.788807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.788773 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08461dce-ee65-4059-9985-ca2af7252bcb-cert\") pod \"odh-model-controller-7998c67dc4-gxlrp\" (UID: \"08461dce-ee65-4059-9985-ca2af7252bcb\") " pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:37.791281 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.791254 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08461dce-ee65-4059-9985-ca2af7252bcb-cert\") pod \"odh-model-controller-7998c67dc4-gxlrp\" (UID: \"08461dce-ee65-4059-9985-ca2af7252bcb\") " pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:37.923360 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:37.923327 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:38.032836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.032682 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ddshn"] Aug 04 00:11:38.037918 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.037891 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:38.040476 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.040451 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Aug 04 00:11:38.040656 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.040614 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-zdrq8\"" Aug 04 00:11:38.045217 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.045160 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ddshn"] Aug 04 00:11:38.089718 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.089648 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gxlrp"] Aug 04 00:11:38.093464 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:11:38.093437 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08461dce_ee65_4059_9985_ca2af7252bcb.slice/crio-7c3d119ca1b760860688cbd708396c79622d509de09f1f4d538473ca1c98e6d4 WatchSource:0}: Error finding container 7c3d119ca1b760860688cbd708396c79622d509de09f1f4d538473ca1c98e6d4: Status 404 returned error can't find the container with id 7c3d119ca1b760860688cbd708396c79622d509de09f1f4d538473ca1c98e6d4 Aug 04 00:11:38.192976 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.192939 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72czm\" (UniqueName: \"kubernetes.io/projected/bbce56ae-aa73-41f1-9e9e-08a3573fc8bd-kube-api-access-72czm\") pod \"model-serving-api-54d89bd5fc-ddshn\" (UID: \"bbce56ae-aa73-41f1-9e9e-08a3573fc8bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:38.193139 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.193002 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbce56ae-aa73-41f1-9e9e-08a3573fc8bd-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddshn\" (UID: \"bbce56ae-aa73-41f1-9e9e-08a3573fc8bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:38.293556 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.293467 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72czm\" (UniqueName: \"kubernetes.io/projected/bbce56ae-aa73-41f1-9e9e-08a3573fc8bd-kube-api-access-72czm\") pod \"model-serving-api-54d89bd5fc-ddshn\" (UID: \"bbce56ae-aa73-41f1-9e9e-08a3573fc8bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:38.293556 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.293531 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbce56ae-aa73-41f1-9e9e-08a3573fc8bd-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddshn\" (UID: \"bbce56ae-aa73-41f1-9e9e-08a3573fc8bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:38.296511 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.296478 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbce56ae-aa73-41f1-9e9e-08a3573fc8bd-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddshn\" (UID: \"bbce56ae-aa73-41f1-9e9e-08a3573fc8bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:38.306832 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.306803 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72czm\" (UniqueName: \"kubernetes.io/projected/bbce56ae-aa73-41f1-9e9e-08a3573fc8bd-kube-api-access-72czm\") pod \"model-serving-api-54d89bd5fc-ddshn\" (UID: \"bbce56ae-aa73-41f1-9e9e-08a3573fc8bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:38.355572 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.355507 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:38.382181 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.381240 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp"] Aug 04 00:11:38.385511 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.384739 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.387769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.387286 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 00:11:38.387769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.387348 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mfbfj\"" Aug 04 00:11:38.387769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.387286 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 00:11:38.387769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.387533 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 00:11:38.388080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.387777 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 00:11:38.396103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.396033 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp"] Aug 04 00:11:38.496237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.495845 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.496237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.495930 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgbm\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-kube-api-access-6cgbm\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.496237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.495967 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.496237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.496005 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.496237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.496034 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cd4c25-c0b9-418e-88e9-3a413bdded8c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.496237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.496064 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.496237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.496098 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.524298 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.524068 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ddshn"] Aug 04 00:11:38.528750 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:11:38.528720 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbce56ae_aa73_41f1_9e9e_08a3573fc8bd.slice/crio-83d8665addbde057a57e66b4516eb2faf2d558168ce337cc6fe9ec5a0a7c49cb WatchSource:0}: Error finding container 83d8665addbde057a57e66b4516eb2faf2d558168ce337cc6fe9ec5a0a7c49cb: Status 404 returned error can't find the container with id 83d8665addbde057a57e66b4516eb2faf2d558168ce337cc6fe9ec5a0a7c49cb Aug 04 00:11:38.596783 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.596701 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.596783 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.596759 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgbm\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-kube-api-access-6cgbm\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.597011 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.596793 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.597011 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.596825 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.597011 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.596851 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cd4c25-c0b9-418e-88e9-3a413bdded8c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.597011 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.596902 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.597011 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.596937 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.597439 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.597385 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.599561 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.599536 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cd4c25-c0b9-418e-88e9-3a413bdded8c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.599682 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.599592 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.599760 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.599740 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.599811 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.599746 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.604485 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.604465 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.604810 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.604789 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgbm\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-kube-api-access-6cgbm\") pod \"istiod-openshift-gateway-7b7995d5cd-ll9gp\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.698677 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.698641 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:38.715944 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.715816 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" event={"ID":"bbce56ae-aa73-41f1-9e9e-08a3573fc8bd","Type":"ContainerStarted","Data":"83d8665addbde057a57e66b4516eb2faf2d558168ce337cc6fe9ec5a0a7c49cb"} Aug 04 00:11:38.720008 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:38.719967 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" event={"ID":"08461dce-ee65-4059-9985-ca2af7252bcb","Type":"ContainerStarted","Data":"7c3d119ca1b760860688cbd708396c79622d509de09f1f4d538473ca1c98e6d4"} Aug 04 00:11:41.731681 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:41.731654 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp"] Aug 04 00:11:41.963756 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:41.963735 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 00:11:41.984583 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:11:41.984553 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cd4c25_c0b9_418e_88e9_3a413bdded8c.slice/crio-e33075e9dba01c1ded3b3c07d2a56b4fb3fd4a4f5e4657944a830bc29c8ffd25 WatchSource:0}: Error finding container e33075e9dba01c1ded3b3c07d2a56b4fb3fd4a4f5e4657944a830bc29c8ffd25: Status 404 returned error can't find the container with id e33075e9dba01c1ded3b3c07d2a56b4fb3fd4a4f5e4657944a830bc29c8ffd25 Aug 04 00:11:42.739058 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:42.739022 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" event={"ID":"f20a59f4-5d4e-4796-b7d8-60b9587e8b3d","Type":"ContainerStarted","Data":"6324903bdf5a29c35abd9ff7a19f68edf7bd628618e0d01f18e3193d4c5f0d83"} Aug 04 00:11:42.740526 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:42.740501 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" event={"ID":"08461dce-ee65-4059-9985-ca2af7252bcb","Type":"ContainerStarted","Data":"9f5fcc138b703bc531e4e917806f4e54c86a2f082539cb5e13a2f3e1dbb0501b"} Aug 04 00:11:42.740648 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:42.740567 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:42.741679 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:42.741652 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" event={"ID":"49cd4c25-c0b9-418e-88e9-3a413bdded8c","Type":"ContainerStarted","Data":"e33075e9dba01c1ded3b3c07d2a56b4fb3fd4a4f5e4657944a830bc29c8ffd25"} Aug 04 00:11:42.743016 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:42.742998 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" event={"ID":"bbce56ae-aa73-41f1-9e9e-08a3573fc8bd","Type":"ContainerStarted","Data":"0eaa4b7ae1ee675f52102683548b8f1e31bdb5d42aaf2541db25ff3ac12b4616"} Aug 04 00:11:42.743118 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:42.743044 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:42.756294 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:42.756258 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-554b4579cb-f2ksc" podStartSLOduration=2.169068555 podStartE2EDuration="6.75624639s" totalImagesPullingTime="4.587177835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:42.755385943 +0000 UTC m=+541.418295423" watchObservedRunningTime="2026-08-04 00:11:42.75624639 +0000 UTC m=+541.419155890" Aug 04 00:11:42.771586 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:42.771542 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" podStartSLOduration=2.844112361 podStartE2EDuration="6.771529437s" totalImagesPullingTime="3.927417076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:42.770304351 +0000 UTC m=+541.433213831" watchObservedRunningTime="2026-08-04 00:11:42.771529437 +0000 UTC m=+541.434438892" Aug 04 00:11:42.786364 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:42.786319 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" podStartSLOduration=1.296290202 podStartE2EDuration="4.786308422s" totalImagesPullingTime="3.49001822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:42.785168263 +0000 UTC m=+541.448077743" watchObservedRunningTime="2026-08-04 00:11:42.786308422 +0000 UTC m=+541.449217899" Aug 04 00:11:45.763687 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:45.763654 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:11:45.763968 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:45.763719 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:11:46.763733 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:46.763696 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" event={"ID":"49cd4c25-c0b9-418e-88e9-3a413bdded8c","Type":"ContainerStarted","Data":"78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1"} Aug 04 00:11:46.764193 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:46.764023 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:46.765811 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:46.765778 2552 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-ll9gp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 00:11:46.765950 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:46.765830 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" podUID="49cd4c25-c0b9-418e-88e9-3a413bdded8c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:11:46.789998 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:46.789944 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" podStartSLOduration=5.041181559 podStartE2EDuration="8.789928291s" totalImagesPullingTime="3.748746732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:46.787309561 +0000 UTC m=+545.450219055" watchObservedRunningTime="2026-08-04 00:11:46.789928291 +0000 UTC m=+545.452837768" Aug 04 00:11:47.361830 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.361810 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw"] Aug 04 00:11:47.364626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.364609 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:11:47.367253 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.367230 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9mxxd\"" Aug 04 00:11:47.367506 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.367486 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Aug 04 00:11:47.367967 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.367949 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpnbz\" (UniqueName: \"kubernetes.io/projected/8e648d96-5199-4788-ab69-79b1e3cad502-kube-api-access-bpnbz\") pod \"kserve-controller-manager-d6f4b65b8-h7qtw\" (UID: \"8e648d96-5199-4788-ab69-79b1e3cad502\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:11:47.368026 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.368001 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e648d96-5199-4788-ab69-79b1e3cad502-cert\") pod \"kserve-controller-manager-d6f4b65b8-h7qtw\" (UID: \"8e648d96-5199-4788-ab69-79b1e3cad502\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:11:47.386938 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.386920 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw"] Aug 04 00:11:47.391299 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.391273 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr"] Aug 04 00:11:47.394482 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.394465 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:47.400165 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.400048 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lwkk9\"" Aug 04 00:11:47.400165 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.400053 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Aug 04 00:11:47.421504 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.421483 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr"] Aug 04 00:11:47.468266 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.468239 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e648d96-5199-4788-ab69-79b1e3cad502-cert\") pod \"kserve-controller-manager-d6f4b65b8-h7qtw\" (UID: \"8e648d96-5199-4788-ab69-79b1e3cad502\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:11:47.468443 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.468283 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe9f0525-cf27-4a1d-9722-698306049b57-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lnpqr\" (UID: \"fe9f0525-cf27-4a1d-9722-698306049b57\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:47.468443 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.468407 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpnbz\" (UniqueName: \"kubernetes.io/projected/8e648d96-5199-4788-ab69-79b1e3cad502-kube-api-access-bpnbz\") pod \"kserve-controller-manager-d6f4b65b8-h7qtw\" (UID: \"8e648d96-5199-4788-ab69-79b1e3cad502\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:11:47.468550 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.468442 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hdq\" (UniqueName: \"kubernetes.io/projected/fe9f0525-cf27-4a1d-9722-698306049b57-kube-api-access-h4hdq\") pod \"llmisvc-controller-manager-777c6bd7d5-lnpqr\" (UID: \"fe9f0525-cf27-4a1d-9722-698306049b57\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:47.470619 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.470600 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e648d96-5199-4788-ab69-79b1e3cad502-cert\") pod \"kserve-controller-manager-d6f4b65b8-h7qtw\" (UID: \"8e648d96-5199-4788-ab69-79b1e3cad502\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:11:47.479587 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.479557 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpnbz\" (UniqueName: \"kubernetes.io/projected/8e648d96-5199-4788-ab69-79b1e3cad502-kube-api-access-bpnbz\") pod \"kserve-controller-manager-d6f4b65b8-h7qtw\" (UID: \"8e648d96-5199-4788-ab69-79b1e3cad502\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:11:47.569113 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.569078 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe9f0525-cf27-4a1d-9722-698306049b57-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lnpqr\" (UID: \"fe9f0525-cf27-4a1d-9722-698306049b57\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:47.569320 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.569129 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hdq\" (UniqueName: \"kubernetes.io/projected/fe9f0525-cf27-4a1d-9722-698306049b57-kube-api-access-h4hdq\") pod \"llmisvc-controller-manager-777c6bd7d5-lnpqr\" (UID: \"fe9f0525-cf27-4a1d-9722-698306049b57\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:47.569320 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:11:47.569236 2552 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Aug 04 00:11:47.569320 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:11:47.569307 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe9f0525-cf27-4a1d-9722-698306049b57-cert podName:fe9f0525-cf27-4a1d-9722-698306049b57 nodeName:}" failed. No retries permitted until 2026-08-04 00:11:48.069287202 +0000 UTC m=+546.732196680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fe9f0525-cf27-4a1d-9722-698306049b57-cert") pod "llmisvc-controller-manager-777c6bd7d5-lnpqr" (UID: "fe9f0525-cf27-4a1d-9722-698306049b57") : secret "llmisvc-webhook-server-cert" not found Aug 04 00:11:47.587430 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.587405 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hdq\" (UniqueName: \"kubernetes.io/projected/fe9f0525-cf27-4a1d-9722-698306049b57-kube-api-access-h4hdq\") pod \"llmisvc-controller-manager-777c6bd7d5-lnpqr\" (UID: \"fe9f0525-cf27-4a1d-9722-698306049b57\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:47.663550 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.663483 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q"] Aug 04 00:11:47.666801 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.666786 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.669801 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.669777 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.669938 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.669815 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4c2ps\"" Aug 04 00:11:47.669938 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.669815 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.669938 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.669877 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26863ba2-e38c-48b9-ae4e-95ec614716ff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.669938 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.669902 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5d5p\" (UniqueName: \"kubernetes.io/projected/26863ba2-e38c-48b9-ae4e-95ec614716ff-kube-api-access-z5d5p\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.669938 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.669933 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.670122 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.669982 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.670122 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.670054 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.670122 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.670072 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.670122 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.670103 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.674048 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.674029 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:11:47.691557 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.691532 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q"] Aug 04 00:11:47.769942 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.769914 2552 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-ll9gp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 00:11:47.770259 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.769985 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" podUID="49cd4c25-c0b9-418e-88e9-3a413bdded8c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:11:47.770505 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.770483 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.770543 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.770518 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.770978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.770642 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.770978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.770749 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.770978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.770918 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.770978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.770974 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.771287 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.771010 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26863ba2-e38c-48b9-ae4e-95ec614716ff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.771287 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.771066 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5d5p\" (UniqueName: \"kubernetes.io/projected/26863ba2-e38c-48b9-ae4e-95ec614716ff-kube-api-access-z5d5p\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.771287 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.771141 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.771287 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.771178 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.771512 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.771295 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.771512 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.771469 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.771623 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.771575 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.772245 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.772219 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26863ba2-e38c-48b9-ae4e-95ec614716ff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.773372 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.773329 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.774032 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.773998 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.780502 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.780477 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5d5p\" (UniqueName: \"kubernetes.io/projected/26863ba2-e38c-48b9-ae4e-95ec614716ff-kube-api-access-z5d5p\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.780873 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.780844 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26863ba2-e38c-48b9-ae4e-95ec614716ff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q\" (UID: \"26863ba2-e38c-48b9-ae4e-95ec614716ff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:47.809069 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.809043 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw"] Aug 04 00:11:47.810487 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:11:47.810465 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e648d96_5199_4788_ab69_79b1e3cad502.slice/crio-b20c9e8f23b0539f282a828ac329cce7e62a434a725d0ef99dbc94b551d2138f WatchSource:0}: Error finding container b20c9e8f23b0539f282a828ac329cce7e62a434a725d0ef99dbc94b551d2138f: Status 404 returned error can't find the container with id b20c9e8f23b0539f282a828ac329cce7e62a434a725d0ef99dbc94b551d2138f Aug 04 00:11:47.977427 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:47.977353 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:48.074080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.073446 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe9f0525-cf27-4a1d-9722-698306049b57-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lnpqr\" (UID: \"fe9f0525-cf27-4a1d-9722-698306049b57\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:48.076351 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.076329 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe9f0525-cf27-4a1d-9722-698306049b57-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lnpqr\" (UID: \"fe9f0525-cf27-4a1d-9722-698306049b57\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:48.111475 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.111449 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q"] Aug 04 00:11:48.112380 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:11:48.112358 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26863ba2_e38c_48b9_ae4e_95ec614716ff.slice/crio-dfcc9c88e9471b363761f933191ab253bfd43766e48c24d9258e39e6c1140a27 WatchSource:0}: Error finding container dfcc9c88e9471b363761f933191ab253bfd43766e48c24d9258e39e6c1140a27: Status 404 returned error can't find the container with id dfcc9c88e9471b363761f933191ab253bfd43766e48c24d9258e39e6c1140a27 Aug 04 00:11:48.303810 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.303731 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:48.434226 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.434200 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr"] Aug 04 00:11:48.435945 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:11:48.435918 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfe9f0525_cf27_4a1d_9722_698306049b57.slice/crio-48de752991eb959065e5562de7931241ac087a191167916d1c58cb976cc5057a WatchSource:0}: Error finding container 48de752991eb959065e5562de7931241ac087a191167916d1c58cb976cc5057a: Status 404 returned error can't find the container with id 48de752991eb959065e5562de7931241ac087a191167916d1c58cb976cc5057a Aug 04 00:11:48.502629 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.502598 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp"] Aug 04 00:11:48.783512 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.783403 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" event={"ID":"26863ba2-e38c-48b9-ae4e-95ec614716ff","Type":"ContainerStarted","Data":"dfcc9c88e9471b363761f933191ab253bfd43766e48c24d9258e39e6c1140a27"} Aug 04 00:11:48.784758 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.784728 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" event={"ID":"fe9f0525-cf27-4a1d-9722-698306049b57","Type":"ContainerStarted","Data":"48de752991eb959065e5562de7931241ac087a191167916d1c58cb976cc5057a"} Aug 04 00:11:48.786131 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.786103 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" event={"ID":"8e648d96-5199-4788-ab69-79b1e3cad502","Type":"ContainerStarted","Data":"b20c9e8f23b0539f282a828ac329cce7e62a434a725d0ef99dbc94b551d2138f"} Aug 04 00:11:48.786254 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.786228 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" podUID="49cd4c25-c0b9-418e-88e9-3a413bdded8c" containerName="discovery" containerID="cri-o://78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1" gracePeriod=30 Aug 04 00:11:48.786987 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:48.786774 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:49.059558 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.059531 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:49.081215 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.081192 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cd4c25-c0b9-418e-88e9-3a413bdded8c-local-certs\") pod \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " Aug 04 00:11:49.081337 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.081232 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cgbm\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-kube-api-access-6cgbm\") pod \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " Aug 04 00:11:49.081337 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.081264 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-token\") pod \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " Aug 04 00:11:49.081337 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.081282 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-dns-cert\") pod \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " Aug 04 00:11:49.081337 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.081311 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-ca-configmap\") pod \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " Aug 04 00:11:49.081337 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.081328 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-kubeconfig\") pod \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " Aug 04 00:11:49.081582 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.081360 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-cacerts\") pod \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\" (UID: \"49cd4c25-c0b9-418e-88e9-3a413bdded8c\") " Aug 04 00:11:49.081849 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.081815 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "49cd4c25-c0b9-418e-88e9-3a413bdded8c" (UID: "49cd4c25-c0b9-418e-88e9-3a413bdded8c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:11:49.083627 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.083573 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "49cd4c25-c0b9-418e-88e9-3a413bdded8c" (UID: "49cd4c25-c0b9-418e-88e9-3a413bdded8c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:11:49.083728 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.083625 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cd4c25-c0b9-418e-88e9-3a413bdded8c-local-certs" (OuterVolumeSpecName: "local-certs") pod "49cd4c25-c0b9-418e-88e9-3a413bdded8c" (UID: "49cd4c25-c0b9-418e-88e9-3a413bdded8c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:11:49.083728 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.083680 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-kube-api-access-6cgbm" (OuterVolumeSpecName: "kube-api-access-6cgbm") pod "49cd4c25-c0b9-418e-88e9-3a413bdded8c" (UID: "49cd4c25-c0b9-418e-88e9-3a413bdded8c"). InnerVolumeSpecName "kube-api-access-6cgbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:11:49.083728 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.083684 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "49cd4c25-c0b9-418e-88e9-3a413bdded8c" (UID: "49cd4c25-c0b9-418e-88e9-3a413bdded8c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:11:49.083842 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.083773 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "49cd4c25-c0b9-418e-88e9-3a413bdded8c" (UID: "49cd4c25-c0b9-418e-88e9-3a413bdded8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:11:49.084010 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.083985 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-cacerts" (OuterVolumeSpecName: "cacerts") pod "49cd4c25-c0b9-418e-88e9-3a413bdded8c" (UID: "49cd4c25-c0b9-418e-88e9-3a413bdded8c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:11:49.182694 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.182659 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:11:49.182694 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.182687 2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-dns-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:11:49.182694 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.182700 2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-csr-ca-configmap\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:11:49.182972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.182710 2552 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-istio-kubeconfig\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:11:49.182972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.182719 2552 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49cd4c25-c0b9-418e-88e9-3a413bdded8c-cacerts\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:11:49.182972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.182727 2552 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cd4c25-c0b9-418e-88e9-3a413bdded8c-local-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:11:49.182972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.182736 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cgbm\" (UniqueName: \"kubernetes.io/projected/49cd4c25-c0b9-418e-88e9-3a413bdded8c-kube-api-access-6cgbm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:11:49.794287 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.794249 2552 generic.go:356] "Generic (PLEG): container finished" podID="49cd4c25-c0b9-418e-88e9-3a413bdded8c" containerID="78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1" exitCode=0 Aug 04 00:11:49.795080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.794305 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" Aug 04 00:11:49.795080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.794339 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" event={"ID":"49cd4c25-c0b9-418e-88e9-3a413bdded8c","Type":"ContainerDied","Data":"78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1"} Aug 04 00:11:49.795080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.794390 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp" event={"ID":"49cd4c25-c0b9-418e-88e9-3a413bdded8c","Type":"ContainerDied","Data":"e33075e9dba01c1ded3b3c07d2a56b4fb3fd4a4f5e4657944a830bc29c8ffd25"} Aug 04 00:11:49.795080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.794419 2552 scope.go:117] "RemoveContainer" containerID="78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1" Aug 04 00:11:49.825588 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.825528 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp"] Aug 04 00:11:49.827445 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.826569 2552 scope.go:117] "RemoveContainer" containerID="78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1" Aug 04 00:11:49.828506 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:11:49.828477 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1\": container with ID starting with 78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1 not found: ID does not exist" containerID="78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1" Aug 04 00:11:49.828612 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.828519 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1"} err="failed to get container status \"78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1\": rpc error: code = NotFound desc = could not find container \"78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1\": container with ID starting with 78a06858dd97fb5b6bf28e1ca87243205736dbb4e921b6e0e3f5a37d309b57b1 not found: ID does not exist" Aug 04 00:11:49.829794 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.829773 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ll9gp"] Aug 04 00:11:49.953226 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:49.953190 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49cd4c25-c0b9-418e-88e9-3a413bdded8c" path="/var/lib/kubelet/pods/49cd4c25-c0b9-418e-88e9-3a413bdded8c/volumes" Aug 04 00:11:50.802111 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:50.802023 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" event={"ID":"8e648d96-5199-4788-ab69-79b1e3cad502","Type":"ContainerStarted","Data":"f8f8fb00d9d157392404b3df25a7a18e03a4db0920052a2230b521164bc97b11"} Aug 04 00:11:50.802544 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:50.802145 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:11:50.819802 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:50.819754 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" podStartSLOduration=1.108590326 podStartE2EDuration="3.819734735s" totalImagesPullingTime="2.711144409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:50.819610062 +0000 UTC m=+549.482519541" watchObservedRunningTime="2026-08-04 00:11:50.819734735 +0000 UTC m=+549.482644214" Aug 04 00:11:52.618522 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:52.618486 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:11:52.618785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:52.618575 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:11:52.618785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:52.618622 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:11:52.813587 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:52.813552 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" event={"ID":"26863ba2-e38c-48b9-ae4e-95ec614716ff","Type":"ContainerStarted","Data":"98b118496bba23b3204928980aa45a9dc274f7690e12b4e5fd475c1366e3ff9d"} Aug 04 00:11:52.815589 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:52.815564 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" event={"ID":"fe9f0525-cf27-4a1d-9722-698306049b57","Type":"ContainerStarted","Data":"561ff81472cee0e5a93ffe36760123428550e74394aa843807b9d44b40543bcb"} Aug 04 00:11:52.815719 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:52.815668 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:11:52.833697 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:52.833651 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" podStartSLOduration=1.329836244 podStartE2EDuration="5.833636529s" totalImagesPullingTime="4.503800285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:52.831252537 +0000 UTC m=+551.494162017" watchObservedRunningTime="2026-08-04 00:11:52.833636529 +0000 UTC m=+551.496546080" Aug 04 00:11:52.847568 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:52.847522 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" podStartSLOduration=1.664481797 podStartE2EDuration="5.847507174s" totalImagesPullingTime="4.183025377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:52.846170581 +0000 UTC m=+551.509080061" watchObservedRunningTime="2026-08-04 00:11:52.847507174 +0000 UTC m=+551.510416651" Aug 04 00:11:52.977721 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:52.977643 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:53.751424 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:53.751394 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-gxlrp" Aug 04 00:11:53.754093 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:53.754072 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ddshn" Aug 04 00:11:53.983001 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:53.982970 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:54.824654 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:54.824613 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:11:54.825673 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:11:54.825654 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q" Aug 04 00:12:21.811795 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:21.811764 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h7qtw" Aug 04 00:12:23.822978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:23.822948 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnpqr" Aug 04 00:12:52.429091 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.429054 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-v2sg7"] Aug 04 00:12:52.429626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.429492 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49cd4c25-c0b9-418e-88e9-3a413bdded8c" containerName="discovery" Aug 04 00:12:52.429626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.429511 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="49cd4c25-c0b9-418e-88e9-3a413bdded8c" containerName="discovery" Aug 04 00:12:52.429626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.429615 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="49cd4c25-c0b9-418e-88e9-3a413bdded8c" containerName="discovery" Aug 04 00:12:52.432407 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.432386 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" Aug 04 00:12:52.435110 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.435090 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 00:12:52.435220 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.435208 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zn5p7\"" Aug 04 00:12:52.435280 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.435244 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 00:12:52.443499 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.443478 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-v2sg7"] Aug 04 00:12:52.491612 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.491583 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5x87\" (UniqueName: \"kubernetes.io/projected/e5d1cd67-c23b-4afc-829c-42b95129c58d-kube-api-access-z5x87\") pod \"authorino-operator-68d88948d8-v2sg7\" (UID: \"e5d1cd67-c23b-4afc-829c-42b95129c58d\") " pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" Aug 04 00:12:52.593318 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.593274 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5x87\" (UniqueName: \"kubernetes.io/projected/e5d1cd67-c23b-4afc-829c-42b95129c58d-kube-api-access-z5x87\") pod \"authorino-operator-68d88948d8-v2sg7\" (UID: \"e5d1cd67-c23b-4afc-829c-42b95129c58d\") " pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" Aug 04 00:12:52.608155 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.608123 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5x87\" (UniqueName: \"kubernetes.io/projected/e5d1cd67-c23b-4afc-829c-42b95129c58d-kube-api-access-z5x87\") pod \"authorino-operator-68d88948d8-v2sg7\" (UID: \"e5d1cd67-c23b-4afc-829c-42b95129c58d\") " pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" Aug 04 00:12:52.742535 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.742442 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" Aug 04 00:12:52.891246 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:52.891222 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-v2sg7"] Aug 04 00:12:52.893414 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:12:52.893386 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d1cd67_c23b_4afc_829c_42b95129c58d.slice/crio-19b6b7f98a19724a822c99939af1df6cf3ee394d5c6ef63e6976354d254f3f72 WatchSource:0}: Error finding container 19b6b7f98a19724a822c99939af1df6cf3ee394d5c6ef63e6976354d254f3f72: Status 404 returned error can't find the container with id 19b6b7f98a19724a822c99939af1df6cf3ee394d5c6ef63e6976354d254f3f72 Aug 04 00:12:53.056930 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:53.056823 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" event={"ID":"e5d1cd67-c23b-4afc-829c-42b95129c58d","Type":"ContainerStarted","Data":"19b6b7f98a19724a822c99939af1df6cf3ee394d5c6ef63e6976354d254f3f72"} Aug 04 00:12:54.625112 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:54.625070 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-fwkml"] Aug 04 00:12:54.634165 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:54.634139 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" Aug 04 00:12:54.636672 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:54.636627 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g2bzx\"" Aug 04 00:12:54.636797 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:54.636637 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 04 00:12:54.641613 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:54.641555 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-fwkml"] Aug 04 00:12:54.708806 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:54.708767 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgj88\" (UniqueName: \"kubernetes.io/projected/8c04aaf8-469f-4662-ab08-d7c7f59fa482-kube-api-access-bgj88\") pod \"dns-operator-controller-manager-5985564776-fwkml\" (UID: \"8c04aaf8-469f-4662-ab08-d7c7f59fa482\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" Aug 04 00:12:54.810305 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:54.810269 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgj88\" (UniqueName: \"kubernetes.io/projected/8c04aaf8-469f-4662-ab08-d7c7f59fa482-kube-api-access-bgj88\") pod \"dns-operator-controller-manager-5985564776-fwkml\" (UID: \"8c04aaf8-469f-4662-ab08-d7c7f59fa482\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" Aug 04 00:12:54.831504 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:54.831477 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgj88\" (UniqueName: \"kubernetes.io/projected/8c04aaf8-469f-4662-ab08-d7c7f59fa482-kube-api-access-bgj88\") pod \"dns-operator-controller-manager-5985564776-fwkml\" (UID: \"8c04aaf8-469f-4662-ab08-d7c7f59fa482\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" Aug 04 00:12:54.946874 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:54.946777 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" Aug 04 00:12:55.520196 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:55.520167 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-fwkml"] Aug 04 00:12:55.522960 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:12:55.522937 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c04aaf8_469f_4662_ab08_d7c7f59fa482.slice/crio-149d250ae1022aaef65b496f53792c77ab17adec4615216fa89f5046edbbfcb7 WatchSource:0}: Error finding container 149d250ae1022aaef65b496f53792c77ab17adec4615216fa89f5046edbbfcb7: Status 404 returned error can't find the container with id 149d250ae1022aaef65b496f53792c77ab17adec4615216fa89f5046edbbfcb7 Aug 04 00:12:56.072464 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:56.072431 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" event={"ID":"8c04aaf8-469f-4662-ab08-d7c7f59fa482","Type":"ContainerStarted","Data":"149d250ae1022aaef65b496f53792c77ab17adec4615216fa89f5046edbbfcb7"} Aug 04 00:12:56.074028 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:56.074003 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" event={"ID":"e5d1cd67-c23b-4afc-829c-42b95129c58d","Type":"ContainerStarted","Data":"055361d8fcc5cce67821575b2b189b80a67e16ccad25da956d3b63e508fa42ec"} Aug 04 00:12:56.074149 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:56.074122 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" Aug 04 00:12:56.102158 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:56.102084 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" podStartSLOduration=1.541738023 podStartE2EDuration="4.102070674s" totalImagesPullingTime="2.560332651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:12:56.100414573 +0000 UTC m=+614.763324052" watchObservedRunningTime="2026-08-04 00:12:56.102070674 +0000 UTC m=+614.764980152" Aug 04 00:12:57.253501 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:57.253470 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx"] Aug 04 00:12:57.256646 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:57.256629 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:12:57.259414 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:57.259384 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-c52zt\"" Aug 04 00:12:57.269605 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:57.269585 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx"] Aug 04 00:12:57.330469 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:57.330442 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zq5\" (UniqueName: \"kubernetes.io/projected/c2df022b-cb19-400f-936b-8b48c6f38979-kube-api-access-q8zq5\") pod \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" (UID: \"c2df022b-cb19-400f-936b-8b48c6f38979\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:12:57.431603 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:57.431575 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zq5\" (UniqueName: \"kubernetes.io/projected/c2df022b-cb19-400f-936b-8b48c6f38979-kube-api-access-q8zq5\") pod \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" (UID: \"c2df022b-cb19-400f-936b-8b48c6f38979\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:12:57.442916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:57.442891 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zq5\" (UniqueName: \"kubernetes.io/projected/c2df022b-cb19-400f-936b-8b48c6f38979-kube-api-access-q8zq5\") pod \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" (UID: \"c2df022b-cb19-400f-936b-8b48c6f38979\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:12:57.567348 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:57.567258 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:12:57.697104 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:57.697080 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx"] Aug 04 00:12:57.699534 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:12:57.699509 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2df022b_cb19_400f_936b_8b48c6f38979.slice/crio-a38adc11b6973379aa3942047d16e0f1378437f2b42a09f441625d7bdf953046 WatchSource:0}: Error finding container a38adc11b6973379aa3942047d16e0f1378437f2b42a09f441625d7bdf953046: Status 404 returned error can't find the container with id a38adc11b6973379aa3942047d16e0f1378437f2b42a09f441625d7bdf953046 Aug 04 00:12:58.088832 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:12:58.088799 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" event={"ID":"c2df022b-cb19-400f-936b-8b48c6f38979","Type":"ContainerStarted","Data":"a38adc11b6973379aa3942047d16e0f1378437f2b42a09f441625d7bdf953046"} Aug 04 00:13:00.099689 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:00.099658 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" event={"ID":"c2df022b-cb19-400f-936b-8b48c6f38979","Type":"ContainerStarted","Data":"ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b"} Aug 04 00:13:00.100122 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:00.099767 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:00.101381 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:00.101348 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" event={"ID":"8c04aaf8-469f-4662-ab08-d7c7f59fa482","Type":"ContainerStarted","Data":"369b95b356b0d72bc8d65559a3fd5270b23b6f27d5cd2a3dd3d916996b73d132"} Aug 04 00:13:00.101510 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:00.101404 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" Aug 04 00:13:00.119745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:00.119701 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" podStartSLOduration=0.960167748 podStartE2EDuration="3.119687534s" totalImagesPullingTime="2.159519786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:13:00.118650979 +0000 UTC m=+618.781560470" watchObservedRunningTime="2026-08-04 00:13:00.119687534 +0000 UTC m=+618.782597011" Aug 04 00:13:00.139920 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:00.139849 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" podStartSLOduration=1.8063888879999999 podStartE2EDuration="6.139834022s" totalImagesPullingTime="4.333445134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:13:00.139360984 +0000 UTC m=+618.802270479" watchObservedRunningTime="2026-08-04 00:13:00.139834022 +0000 UTC m=+618.802743500" Aug 04 00:13:01.394232 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.394193 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7"] Aug 04 00:13:01.397664 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.397642 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:01.400665 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.400647 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tmg2q\"" Aug 04 00:13:01.414617 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.414593 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7"] Aug 04 00:13:01.463664 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.463636 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ndm\" (UniqueName: \"kubernetes.io/projected/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-kube-api-access-n9ndm\") pod \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" (UID: \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:01.463789 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.463726 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" (UID: \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:01.564944 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.564918 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" (UID: \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:01.565047 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.564964 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ndm\" (UniqueName: \"kubernetes.io/projected/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-kube-api-access-n9ndm\") pod \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" (UID: \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:01.565296 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.565279 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" (UID: \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:01.574001 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.573983 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ndm\" (UniqueName: \"kubernetes.io/projected/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-kube-api-access-n9ndm\") pod \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" (UID: \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:01.706964 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.706892 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:01.835975 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:01.835902 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7"] Aug 04 00:13:01.838692 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:13:01.838662 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode635b7d9_73e3_48d8_bca4_0606d5d3a9f3.slice/crio-d614ec5d09a58619d761bd675597d242b7dd0dbe2d2e6528bbafc0d4744ab83b WatchSource:0}: Error finding container d614ec5d09a58619d761bd675597d242b7dd0dbe2d2e6528bbafc0d4744ab83b: Status 404 returned error can't find the container with id d614ec5d09a58619d761bd675597d242b7dd0dbe2d2e6528bbafc0d4744ab83b Aug 04 00:13:02.118600 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:02.118566 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" event={"ID":"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3","Type":"ContainerStarted","Data":"d614ec5d09a58619d761bd675597d242b7dd0dbe2d2e6528bbafc0d4744ab83b"} Aug 04 00:13:06.142260 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:06.142225 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" event={"ID":"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3","Type":"ContainerStarted","Data":"525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e"} Aug 04 00:13:06.142692 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:06.142374 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:06.161364 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:06.161316 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" podStartSLOduration=0.952095202 podStartE2EDuration="5.161301599s" totalImagesPullingTime="4.209206397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:13:06.159808645 +0000 UTC m=+624.822718133" watchObservedRunningTime="2026-08-04 00:13:06.161301599 +0000 UTC m=+624.824211076" Aug 04 00:13:07.080295 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:07.080266 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-v2sg7" Aug 04 00:13:11.107912 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:11.107883 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-fwkml" Aug 04 00:13:11.108261 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:11.107961 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:17.148529 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:17.148502 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:18.179715 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.179684 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7"] Aug 04 00:13:18.185798 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.185779 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:18.195704 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.195679 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7"] Aug 04 00:13:18.301670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.301635 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjnrh\" (UniqueName: \"kubernetes.io/projected/5a7069b5-2f86-4b63-b314-4ca71705188b-kube-api-access-vjnrh\") pod \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" (UID: \"5a7069b5-2f86-4b63-b314-4ca71705188b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:18.301840 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.301692 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a7069b5-2f86-4b63-b314-4ca71705188b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" (UID: \"5a7069b5-2f86-4b63-b314-4ca71705188b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:18.402126 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.402099 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a7069b5-2f86-4b63-b314-4ca71705188b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" (UID: \"5a7069b5-2f86-4b63-b314-4ca71705188b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:18.402251 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.402159 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjnrh\" (UniqueName: \"kubernetes.io/projected/5a7069b5-2f86-4b63-b314-4ca71705188b-kube-api-access-vjnrh\") pod \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" (UID: \"5a7069b5-2f86-4b63-b314-4ca71705188b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:18.402431 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.402414 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a7069b5-2f86-4b63-b314-4ca71705188b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" (UID: \"5a7069b5-2f86-4b63-b314-4ca71705188b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:18.412968 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.412942 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjnrh\" (UniqueName: \"kubernetes.io/projected/5a7069b5-2f86-4b63-b314-4ca71705188b-kube-api-access-vjnrh\") pod \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" (UID: \"5a7069b5-2f86-4b63-b314-4ca71705188b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:18.496795 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.496720 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:18.642160 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.642139 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7"] Aug 04 00:13:18.644297 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:13:18.644263 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7069b5_2f86_4b63_b314_4ca71705188b.slice/crio-52f3e30a4edba8c1d2287a48c94178069c64d9c88c38debb07e8e2eb3b0f9271 WatchSource:0}: Error finding container 52f3e30a4edba8c1d2287a48c94178069c64d9c88c38debb07e8e2eb3b0f9271: Status 404 returned error can't find the container with id 52f3e30a4edba8c1d2287a48c94178069c64d9c88c38debb07e8e2eb3b0f9271 Aug 04 00:13:18.874715 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.874681 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7"] Aug 04 00:13:18.874928 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.874906 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" containerName="manager" containerID="cri-o://525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e" gracePeriod=2 Aug 04 00:13:18.877290 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:18.877256 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:18.879414 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.879392 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7"] Aug 04 00:13:18.893613 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.893586 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7"] Aug 04 00:13:18.897465 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.897429 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr"] Aug 04 00:13:18.897800 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.897787 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" containerName="manager" Aug 04 00:13:18.897800 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.897801 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" containerName="manager" Aug 04 00:13:18.897951 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.897884 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" containerName="manager" Aug 04 00:13:18.911536 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.911495 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7"] Aug 04 00:13:18.911895 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.911849 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:18.913686 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.913662 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr"] Aug 04 00:13:18.919252 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.919230 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql"] Aug 04 00:13:18.919702 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.919685 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" containerName="manager" Aug 04 00:13:18.919752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.919708 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" containerName="manager" Aug 04 00:13:18.919820 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.919808 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" containerName="manager" Aug 04 00:13:18.923353 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.923325 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx"] Aug 04 00:13:18.923654 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.923539 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:18.923754 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.923705 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" containerName="manager" containerID="cri-o://ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b" gracePeriod=2 Aug 04 00:13:18.924503 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.924471 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx"] Aug 04 00:13:18.937431 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:18.937196 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:18.939322 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:18.939287 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:18.941558 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.941535 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql"] Aug 04 00:13:18.947756 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.947732 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm"] Aug 04 00:13:18.948063 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.948049 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" containerName="manager" Aug 04 00:13:18.948063 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.948064 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" containerName="manager" Aug 04 00:13:18.948182 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.948113 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" containerName="manager" Aug 04 00:13:18.951207 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.951191 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" Aug 04 00:13:18.962366 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:18.962344 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:18.962461 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:18.962422 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm"] Aug 04 00:13:18.964189 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:18.964142 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:18.992113 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:18.992075 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:19.008635 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.008613 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7b5r\" (UniqueName: \"kubernetes.io/projected/63cfe342-44f8-4c23-8ad7-b1aa22b1a943-kube-api-access-m7b5r\") pod \"limitador-operator-controller-manager-6c549b6d85-wdpgm\" (UID: \"63cfe342-44f8-4c23-8ad7-b1aa22b1a943\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" Aug 04 00:13:19.008747 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.008645 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tfq\" (UniqueName: \"kubernetes.io/projected/8f2e26fb-0213-4c11-82ff-989a366ecef9-kube-api-access-l7tfq\") pod \"kuadrant-operator-controller-manager-96db98c47-99tql\" (UID: \"8f2e26fb-0213-4c11-82ff-989a366ecef9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:19.008747 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.008671 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-4sjhr\" (UID: \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:19.008853 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.008763 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f2e26fb-0213-4c11-82ff-989a366ecef9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-99tql\" (UID: \"8f2e26fb-0213-4c11-82ff-989a366ecef9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:19.008853 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.008786 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxqkk\" (UniqueName: \"kubernetes.io/projected/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-kube-api-access-rxqkk\") pod \"kuadrant-operator-controller-manager-58fc56757c-4sjhr\" (UID: \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:19.109324 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.109293 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f2e26fb-0213-4c11-82ff-989a366ecef9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-99tql\" (UID: \"8f2e26fb-0213-4c11-82ff-989a366ecef9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:19.109465 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.109326 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxqkk\" (UniqueName: \"kubernetes.io/projected/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-kube-api-access-rxqkk\") pod \"kuadrant-operator-controller-manager-58fc56757c-4sjhr\" (UID: \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:19.109465 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.109416 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7b5r\" (UniqueName: \"kubernetes.io/projected/63cfe342-44f8-4c23-8ad7-b1aa22b1a943-kube-api-access-m7b5r\") pod \"limitador-operator-controller-manager-6c549b6d85-wdpgm\" (UID: \"63cfe342-44f8-4c23-8ad7-b1aa22b1a943\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" Aug 04 00:13:19.109465 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.109441 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tfq\" (UniqueName: \"kubernetes.io/projected/8f2e26fb-0213-4c11-82ff-989a366ecef9-kube-api-access-l7tfq\") pod \"kuadrant-operator-controller-manager-96db98c47-99tql\" (UID: \"8f2e26fb-0213-4c11-82ff-989a366ecef9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:19.109629 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.109484 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-4sjhr\" (UID: \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:19.109729 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.109708 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f2e26fb-0213-4c11-82ff-989a366ecef9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-99tql\" (UID: \"8f2e26fb-0213-4c11-82ff-989a366ecef9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:19.109835 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.109819 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-4sjhr\" (UID: \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:19.122779 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.122741 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7b5r\" (UniqueName: \"kubernetes.io/projected/63cfe342-44f8-4c23-8ad7-b1aa22b1a943-kube-api-access-m7b5r\") pod \"limitador-operator-controller-manager-6c549b6d85-wdpgm\" (UID: \"63cfe342-44f8-4c23-8ad7-b1aa22b1a943\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" Aug 04 00:13:19.122931 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.122741 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tfq\" (UniqueName: \"kubernetes.io/projected/8f2e26fb-0213-4c11-82ff-989a366ecef9-kube-api-access-l7tfq\") pod \"kuadrant-operator-controller-manager-96db98c47-99tql\" (UID: \"8f2e26fb-0213-4c11-82ff-989a366ecef9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:19.122998 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.122933 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxqkk\" (UniqueName: \"kubernetes.io/projected/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-kube-api-access-rxqkk\") pod \"kuadrant-operator-controller-manager-58fc56757c-4sjhr\" (UID: \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:19.150842 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.150748 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:19.153037 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.153011 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:19.154780 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.154749 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:19.157707 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.157687 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:19.159665 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.159643 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:19.161525 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.161505 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:19.200917 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.200884 2552 generic.go:356] "Generic (PLEG): container finished" podID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" containerID="525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e" exitCode=0 Aug 04 00:13:19.200917 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.200921 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:19.201347 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.200979 2552 scope.go:117] "RemoveContainer" containerID="525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e" Aug 04 00:13:19.202833 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.202818 2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" container="manager" Aug 04 00:13:19.204109 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.204088 2552 generic.go:356] "Generic (PLEG): container finished" podID="c2df022b-cb19-400f-936b-8b48c6f38979" containerID="ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b" exitCode=0 Aug 04 00:13:19.204209 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.204125 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:19.204349 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.204324 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:19.206209 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.206188 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:19.208052 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.208031 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:19.209802 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.209778 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:19.209981 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.209909 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9ndm\" (UniqueName: \"kubernetes.io/projected/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-kube-api-access-n9ndm\") pod \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\" (UID: \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\") " Aug 04 00:13:19.209981 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.209943 2552 scope.go:117] "RemoveContainer" containerID="525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e" Aug 04 00:13:19.210086 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.210010 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8zq5\" (UniqueName: \"kubernetes.io/projected/c2df022b-cb19-400f-936b-8b48c6f38979-kube-api-access-q8zq5\") pod \"c2df022b-cb19-400f-936b-8b48c6f38979\" (UID: \"c2df022b-cb19-400f-936b-8b48c6f38979\") " Aug 04 00:13:19.210086 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.210080 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-extensions-socket-volume\") pod \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\" (UID: \"e635b7d9-73e3-48d8-bca4-0606d5d3a9f3\") " Aug 04 00:13:19.210257 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.210233 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e\": container with ID starting with 525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e not found: ID does not exist" containerID="525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e" Aug 04 00:13:19.210316 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.210266 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e"} err="failed to get container status \"525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e\": rpc error: code = NotFound desc = could not find container \"525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e\": container with ID starting with 525620a5522224140da3c1c2c69c03316af9666b203981290e81754df5face2e not found: ID does not exist" Aug 04 00:13:19.210316 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.210284 2552 scope.go:117] "RemoveContainer" containerID="ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b" Aug 04 00:13:19.210926 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.210897 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" (UID: "e635b7d9-73e3-48d8-bca4-0606d5d3a9f3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:13:19.211837 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.211805 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:19.212136 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.212114 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-kube-api-access-n9ndm" (OuterVolumeSpecName: "kube-api-access-n9ndm") pod "e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" (UID: "e635b7d9-73e3-48d8-bca4-0606d5d3a9f3"). InnerVolumeSpecName "kube-api-access-n9ndm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:13:19.212228 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.212202 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2df022b-cb19-400f-936b-8b48c6f38979-kube-api-access-q8zq5" (OuterVolumeSpecName: "kube-api-access-q8zq5") pod "c2df022b-cb19-400f-936b-8b48c6f38979" (UID: "c2df022b-cb19-400f-936b-8b48c6f38979"). InnerVolumeSpecName "kube-api-access-q8zq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:13:19.217829 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.217814 2552 scope.go:117] "RemoveContainer" containerID="ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b" Aug 04 00:13:19.218082 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.218065 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b\": container with ID starting with ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b not found: ID does not exist" containerID="ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b" Aug 04 00:13:19.218133 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.218087 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b"} err="failed to get container status \"ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b\": rpc error: code = NotFound desc = could not find container \"ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b\": container with ID starting with ee963d5de520ad61c47b23a0bd593e9498843aac81a75f1d4c33a07324ffa36b not found: ID does not exist" Aug 04 00:13:19.280174 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.280131 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:19.290963 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.290946 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:19.297111 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.297093 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" Aug 04 00:13:19.311167 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.311139 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8zq5\" (UniqueName: \"kubernetes.io/projected/c2df022b-cb19-400f-936b-8b48c6f38979-kube-api-access-q8zq5\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:13:19.311167 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.311166 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-extensions-socket-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:13:19.311328 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.311178 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9ndm\" (UniqueName: \"kubernetes.io/projected/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3-kube-api-access-n9ndm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:13:19.435997 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.435964 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr"] Aug 04 00:13:19.439029 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:13:19.438994 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47e1b04_e2f4_45da_849e_b180d9dfa3b1.slice/crio-d5388c1acdc8c9d3484d340ad65248d9ae9e59684088698dac09c898f5f838b9 WatchSource:0}: Error finding container d5388c1acdc8c9d3484d340ad65248d9ae9e59684088698dac09c898f5f838b9: Status 404 returned error can't find the container with id d5388c1acdc8c9d3484d340ad65248d9ae9e59684088698dac09c898f5f838b9 Aug 04 00:13:19.512090 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.512061 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:19.513987 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.513961 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:19.515930 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.515905 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:19.518100 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.518078 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-7g6w7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7g6w7" Aug 04 00:13:19.519898 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.519852 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-rmpqx\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rmpqx" Aug 04 00:13:19.521551 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:19.521534 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:19.679190 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.679141 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm"] Aug 04 00:13:19.681337 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.681304 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql"] Aug 04 00:13:19.681601 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:13:19.681579 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2e26fb_0213_4c11_82ff_989a366ecef9.slice/crio-a621857829ead6ac3212d3493164a82ebdcc3c83555ca211198c038e2a279da5 WatchSource:0}: Error finding container a621857829ead6ac3212d3493164a82ebdcc3c83555ca211198c038e2a279da5: Status 404 returned error can't find the container with id a621857829ead6ac3212d3493164a82ebdcc3c83555ca211198c038e2a279da5 Aug 04 00:13:19.682387 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:13:19.682367 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cfe342_44f8_4c23_8ad7_b1aa22b1a943.slice/crio-7cae65449ef14367c928956159c06fc9a972a98cfd5b2ef6d46469cb1ef51f0a WatchSource:0}: Error finding container 7cae65449ef14367c928956159c06fc9a972a98cfd5b2ef6d46469cb1ef51f0a: Status 404 returned error can't find the container with id 7cae65449ef14367c928956159c06fc9a972a98cfd5b2ef6d46469cb1ef51f0a Aug 04 00:13:19.952728 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.952656 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2df022b-cb19-400f-936b-8b48c6f38979" path="/var/lib/kubelet/pods/c2df022b-cb19-400f-936b-8b48c6f38979/volumes" Aug 04 00:13:19.953052 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:19.953037 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e635b7d9-73e3-48d8-bca4-0606d5d3a9f3" path="/var/lib/kubelet/pods/e635b7d9-73e3-48d8-bca4-0606d5d3a9f3/volumes" Aug 04 00:13:20.211113 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.211023 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" event={"ID":"8f2e26fb-0213-4c11-82ff-989a366ecef9","Type":"ContainerStarted","Data":"abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e"} Aug 04 00:13:20.211113 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.211082 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" event={"ID":"8f2e26fb-0213-4c11-82ff-989a366ecef9","Type":"ContainerStarted","Data":"a621857829ead6ac3212d3493164a82ebdcc3c83555ca211198c038e2a279da5"} Aug 04 00:13:20.211566 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.211126 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:20.213461 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.213442 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" event={"ID":"63cfe342-44f8-4c23-8ad7-b1aa22b1a943","Type":"ContainerStarted","Data":"bc569bb31cefc284bf3668ad7e5c1135b6269623d35408ad46bf04b5a87d3a2b"} Aug 04 00:13:20.213544 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.213482 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" event={"ID":"63cfe342-44f8-4c23-8ad7-b1aa22b1a943","Type":"ContainerStarted","Data":"7cae65449ef14367c928956159c06fc9a972a98cfd5b2ef6d46469cb1ef51f0a"} Aug 04 00:13:20.213618 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.213557 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" Aug 04 00:13:20.215134 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.215108 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" containerName="manager" containerID="cri-o://70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7" gracePeriod=2 Aug 04 00:13:20.216729 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.216708 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" event={"ID":"d47e1b04-e2f4-45da-849e-b180d9dfa3b1","Type":"ContainerStarted","Data":"786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2"} Aug 04 00:13:20.216813 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.216740 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" event={"ID":"d47e1b04-e2f4-45da-849e-b180d9dfa3b1","Type":"ContainerStarted","Data":"d5388c1acdc8c9d3484d340ad65248d9ae9e59684088698dac09c898f5f838b9"} Aug 04 00:13:20.216884 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.216830 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:20.259846 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.259798 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" podStartSLOduration=2.259782564 podStartE2EDuration="2.259782564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:13:20.25734472 +0000 UTC m=+638.920254198" watchObservedRunningTime="2026-08-04 00:13:20.259782564 +0000 UTC m=+638.922692042" Aug 04 00:13:20.299100 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.299039 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" podStartSLOduration=2.299020922 podStartE2EDuration="2.299020922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:13:20.297901461 +0000 UTC m=+638.960810941" watchObservedRunningTime="2026-08-04 00:13:20.299020922 +0000 UTC m=+638.961930399" Aug 04 00:13:20.337730 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.337674 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" podStartSLOduration=2.33765296 podStartE2EDuration="2.33765296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:13:20.335696991 +0000 UTC m=+638.998606469" watchObservedRunningTime="2026-08-04 00:13:20.33765296 +0000 UTC m=+639.000562439" Aug 04 00:13:20.460585 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.460564 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:20.462653 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:20.462602 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:20.524051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.524019 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjnrh\" (UniqueName: \"kubernetes.io/projected/5a7069b5-2f86-4b63-b314-4ca71705188b-kube-api-access-vjnrh\") pod \"5a7069b5-2f86-4b63-b314-4ca71705188b\" (UID: \"5a7069b5-2f86-4b63-b314-4ca71705188b\") " Aug 04 00:13:20.524203 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.524091 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a7069b5-2f86-4b63-b314-4ca71705188b-extensions-socket-volume\") pod \"5a7069b5-2f86-4b63-b314-4ca71705188b\" (UID: \"5a7069b5-2f86-4b63-b314-4ca71705188b\") " Aug 04 00:13:20.524333 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.524307 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7069b5-2f86-4b63-b314-4ca71705188b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5a7069b5-2f86-4b63-b314-4ca71705188b" (UID: "5a7069b5-2f86-4b63-b314-4ca71705188b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:13:20.526063 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.526040 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7069b5-2f86-4b63-b314-4ca71705188b-kube-api-access-vjnrh" (OuterVolumeSpecName: "kube-api-access-vjnrh") pod "5a7069b5-2f86-4b63-b314-4ca71705188b" (UID: "5a7069b5-2f86-4b63-b314-4ca71705188b"). InnerVolumeSpecName "kube-api-access-vjnrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:13:20.624728 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.624698 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a7069b5-2f86-4b63-b314-4ca71705188b-extensions-socket-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:13:20.624728 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:20.624725 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjnrh\" (UniqueName: \"kubernetes.io/projected/5a7069b5-2f86-4b63-b314-4ca71705188b-kube-api-access-vjnrh\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:13:21.223320 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:21.223282 2552 generic.go:356] "Generic (PLEG): container finished" podID="5a7069b5-2f86-4b63-b314-4ca71705188b" containerID="70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7" exitCode=2 Aug 04 00:13:21.223791 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:21.223350 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:21.223791 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:21.223390 2552 scope.go:117] "RemoveContainer" containerID="70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7" Aug 04 00:13:21.225847 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:21.225818 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:21.233752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:21.233733 2552 scope.go:117] "RemoveContainer" containerID="70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7" Aug 04 00:13:21.234113 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:21.234086 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:21.234236 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:21.234191 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7\": container with ID starting with 70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7 not found: ID does not exist" containerID="70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7" Aug 04 00:13:21.234236 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:21.234217 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7"} err="failed to get container status \"70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7\": rpc error: code = NotFound desc = could not find container \"70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7\": container with ID starting with 70cfde36265e8479e6024900f2129ec10565e337deb8c544e66289d87ac249e7 not found: ID does not exist" Aug 04 00:13:21.950592 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:21.950548 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zxcl7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zxcl7" Aug 04 00:13:21.953478 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:21.953452 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a7069b5-2f86-4b63-b314-4ca71705188b" path="/var/lib/kubelet/pods/5a7069b5-2f86-4b63-b314-4ca71705188b/volumes" Aug 04 00:13:31.226505 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.226471 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:31.227046 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.226551 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wdpgm" Aug 04 00:13:31.227046 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.226587 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:31.293152 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.293114 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr"] Aug 04 00:13:31.295131 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.295087 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" podUID="d47e1b04-e2f4-45da-849e-b180d9dfa3b1" containerName="manager" containerID="cri-o://786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2" gracePeriod=10 Aug 04 00:13:31.546074 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.546048 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:31.586384 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.586355 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78"] Aug 04 00:13:31.586770 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.586745 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47e1b04-e2f4-45da-849e-b180d9dfa3b1" containerName="manager" Aug 04 00:13:31.586770 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.586770 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47e1b04-e2f4-45da-849e-b180d9dfa3b1" containerName="manager" Aug 04 00:13:31.587030 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.586891 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d47e1b04-e2f4-45da-849e-b180d9dfa3b1" containerName="manager" Aug 04 00:13:31.588975 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.588952 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:31.602208 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.602187 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78"] Aug 04 00:13:31.618563 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.618545 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxqkk\" (UniqueName: \"kubernetes.io/projected/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-kube-api-access-rxqkk\") pod \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\" (UID: \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\") " Aug 04 00:13:31.618662 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.618628 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-extensions-socket-volume\") pod \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\" (UID: \"d47e1b04-e2f4-45da-849e-b180d9dfa3b1\") " Aug 04 00:13:31.618981 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.618956 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d47e1b04-e2f4-45da-849e-b180d9dfa3b1" (UID: "d47e1b04-e2f4-45da-849e-b180d9dfa3b1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:13:31.620690 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.620670 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-kube-api-access-rxqkk" (OuterVolumeSpecName: "kube-api-access-rxqkk") pod "d47e1b04-e2f4-45da-849e-b180d9dfa3b1" (UID: "d47e1b04-e2f4-45da-849e-b180d9dfa3b1"). InnerVolumeSpecName "kube-api-access-rxqkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:13:31.719767 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.719736 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28baae9-478a-476d-b8d3-b1e3c876ef44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-wzp78\" (UID: \"a28baae9-478a-476d-b8d3-b1e3c876ef44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:31.719975 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.719805 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5t6m\" (UniqueName: \"kubernetes.io/projected/a28baae9-478a-476d-b8d3-b1e3c876ef44-kube-api-access-w5t6m\") pod \"kuadrant-operator-controller-manager-68cf589589-wzp78\" (UID: \"a28baae9-478a-476d-b8d3-b1e3c876ef44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:31.719975 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.719928 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-extensions-socket-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:13:31.719975 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.719950 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxqkk\" (UniqueName: \"kubernetes.io/projected/d47e1b04-e2f4-45da-849e-b180d9dfa3b1-kube-api-access-rxqkk\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:13:31.820843 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.820766 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5t6m\" (UniqueName: \"kubernetes.io/projected/a28baae9-478a-476d-b8d3-b1e3c876ef44-kube-api-access-w5t6m\") pod \"kuadrant-operator-controller-manager-68cf589589-wzp78\" (UID: \"a28baae9-478a-476d-b8d3-b1e3c876ef44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:31.820843 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.820810 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28baae9-478a-476d-b8d3-b1e3c876ef44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-wzp78\" (UID: \"a28baae9-478a-476d-b8d3-b1e3c876ef44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:31.821217 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.821196 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28baae9-478a-476d-b8d3-b1e3c876ef44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-wzp78\" (UID: \"a28baae9-478a-476d-b8d3-b1e3c876ef44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:31.829404 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.829384 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5t6m\" (UniqueName: \"kubernetes.io/projected/a28baae9-478a-476d-b8d3-b1e3c876ef44-kube-api-access-w5t6m\") pod \"kuadrant-operator-controller-manager-68cf589589-wzp78\" (UID: \"a28baae9-478a-476d-b8d3-b1e3c876ef44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:31.901643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:31.901617 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:32.037914 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.037891 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78"] Aug 04 00:13:32.040243 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:13:32.040215 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28baae9_478a_476d_b8d3_b1e3c876ef44.slice/crio-0fbb529197795eedd2d32c73408bd3e700fa0cceadc281e5f3be1a77877b492f WatchSource:0}: Error finding container 0fbb529197795eedd2d32c73408bd3e700fa0cceadc281e5f3be1a77877b492f: Status 404 returned error can't find the container with id 0fbb529197795eedd2d32c73408bd3e700fa0cceadc281e5f3be1a77877b492f Aug 04 00:13:32.278286 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.278253 2552 generic.go:356] "Generic (PLEG): container finished" podID="d47e1b04-e2f4-45da-849e-b180d9dfa3b1" containerID="786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2" exitCode=0 Aug 04 00:13:32.278732 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.278314 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" Aug 04 00:13:32.278732 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.278328 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" event={"ID":"d47e1b04-e2f4-45da-849e-b180d9dfa3b1","Type":"ContainerDied","Data":"786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2"} Aug 04 00:13:32.278732 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.278388 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr" event={"ID":"d47e1b04-e2f4-45da-849e-b180d9dfa3b1","Type":"ContainerDied","Data":"d5388c1acdc8c9d3484d340ad65248d9ae9e59684088698dac09c898f5f838b9"} Aug 04 00:13:32.278732 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.278411 2552 scope.go:117] "RemoveContainer" containerID="786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2" Aug 04 00:13:32.280273 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.280220 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" event={"ID":"a28baae9-478a-476d-b8d3-b1e3c876ef44","Type":"ContainerStarted","Data":"1160bc437f3b9a70dd4fb88eea10febe750961b0da4f28e6dc1268042669b796"} Aug 04 00:13:32.280393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.280281 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:32.280393 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.280305 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" event={"ID":"a28baae9-478a-476d-b8d3-b1e3c876ef44","Type":"ContainerStarted","Data":"0fbb529197795eedd2d32c73408bd3e700fa0cceadc281e5f3be1a77877b492f"} Aug 04 00:13:32.287178 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.287150 2552 scope.go:117] "RemoveContainer" containerID="786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2" Aug 04 00:13:32.287452 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:32.287431 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2\": container with ID starting with 786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2 not found: ID does not exist" containerID="786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2" Aug 04 00:13:32.287527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.287459 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2"} err="failed to get container status \"786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2\": rpc error: code = NotFound desc = could not find container \"786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2\": container with ID starting with 786ee58e7b9a7cd86d66824862e350454c775ec7224424cfbdcd41b63f52dae2 not found: ID does not exist" Aug 04 00:13:32.303682 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.303635 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" podStartSLOduration=1.303621747 podStartE2EDuration="1.303621747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:13:32.301363419 +0000 UTC m=+650.964272910" watchObservedRunningTime="2026-08-04 00:13:32.303621747 +0000 UTC m=+650.966531226" Aug 04 00:13:32.317679 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.317653 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr"] Aug 04 00:13:32.325492 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:32.325471 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-4sjhr"] Aug 04 00:13:33.953083 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:33.953045 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d47e1b04-e2f4-45da-849e-b180d9dfa3b1" path="/var/lib/kubelet/pods/d47e1b04-e2f4-45da-849e-b180d9dfa3b1/volumes" Aug 04 00:13:43.287955 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.287921 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:13:43.342867 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.342833 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql"] Aug 04 00:13:43.343084 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.343060 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" podUID="8f2e26fb-0213-4c11-82ff-989a366ecef9" containerName="manager" containerID="cri-o://abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e" gracePeriod=10 Aug 04 00:13:43.588819 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.588797 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:43.716569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.716543 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f2e26fb-0213-4c11-82ff-989a366ecef9-extensions-socket-volume\") pod \"8f2e26fb-0213-4c11-82ff-989a366ecef9\" (UID: \"8f2e26fb-0213-4c11-82ff-989a366ecef9\") " Aug 04 00:13:43.716732 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.716604 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7tfq\" (UniqueName: \"kubernetes.io/projected/8f2e26fb-0213-4c11-82ff-989a366ecef9-kube-api-access-l7tfq\") pod \"8f2e26fb-0213-4c11-82ff-989a366ecef9\" (UID: \"8f2e26fb-0213-4c11-82ff-989a366ecef9\") " Aug 04 00:13:43.717058 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.717017 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2e26fb-0213-4c11-82ff-989a366ecef9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8f2e26fb-0213-4c11-82ff-989a366ecef9" (UID: "8f2e26fb-0213-4c11-82ff-989a366ecef9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:13:43.718726 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.718702 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2e26fb-0213-4c11-82ff-989a366ecef9-kube-api-access-l7tfq" (OuterVolumeSpecName: "kube-api-access-l7tfq") pod "8f2e26fb-0213-4c11-82ff-989a366ecef9" (UID: "8f2e26fb-0213-4c11-82ff-989a366ecef9"). InnerVolumeSpecName "kube-api-access-l7tfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:13:43.817132 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.817108 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f2e26fb-0213-4c11-82ff-989a366ecef9-extensions-socket-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:13:43.817132 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:43.817128 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7tfq\" (UniqueName: \"kubernetes.io/projected/8f2e26fb-0213-4c11-82ff-989a366ecef9-kube-api-access-l7tfq\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:13:44.344509 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:44.344427 2552 generic.go:356] "Generic (PLEG): container finished" podID="8f2e26fb-0213-4c11-82ff-989a366ecef9" containerID="abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e" exitCode=0 Aug 04 00:13:44.344509 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:44.344473 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" Aug 04 00:13:44.345028 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:44.344526 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" event={"ID":"8f2e26fb-0213-4c11-82ff-989a366ecef9","Type":"ContainerDied","Data":"abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e"} Aug 04 00:13:44.345028 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:44.344583 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql" event={"ID":"8f2e26fb-0213-4c11-82ff-989a366ecef9","Type":"ContainerDied","Data":"a621857829ead6ac3212d3493164a82ebdcc3c83555ca211198c038e2a279da5"} Aug 04 00:13:44.345028 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:44.344617 2552 scope.go:117] "RemoveContainer" containerID="abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e" Aug 04 00:13:44.352735 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:44.352720 2552 scope.go:117] "RemoveContainer" containerID="abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e" Aug 04 00:13:44.352978 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:13:44.352959 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e\": container with ID starting with abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e not found: ID does not exist" containerID="abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e" Aug 04 00:13:44.353034 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:44.352985 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e"} err="failed to get container status \"abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e\": rpc error: code = NotFound desc = could not find container \"abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e\": container with ID starting with abde9e20f98ed00a251d7efd939e535bff0fe264c0e22e34e7d243218662201e not found: ID does not exist" Aug 04 00:13:44.363927 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:44.363904 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql"] Aug 04 00:13:44.369381 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:44.369360 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-99tql"] Aug 04 00:13:45.952367 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:45.952333 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2e26fb-0213-4c11-82ff-989a366ecef9" path="/var/lib/kubelet/pods/8f2e26fb-0213-4c11-82ff-989a366ecef9/volumes" Aug 04 00:13:47.581444 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.581415 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8"] Aug 04 00:13:47.581790 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.581738 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2e26fb-0213-4c11-82ff-989a366ecef9" containerName="manager" Aug 04 00:13:47.581790 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.581748 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2e26fb-0213-4c11-82ff-989a366ecef9" containerName="manager" Aug 04 00:13:47.581883 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.581811 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2e26fb-0213-4c11-82ff-989a366ecef9" containerName="manager" Aug 04 00:13:47.586058 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.586038 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.589243 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.589225 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dmk9j\"" Aug 04 00:13:47.608774 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.608749 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8"] Aug 04 00:13:47.755315 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.755280 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.755486 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.755319 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8s2\" (UniqueName: \"kubernetes.io/projected/9c4e0926-157d-4f36-874d-bf2af64dbc79-kube-api-access-lt8s2\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.755486 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.755348 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.755486 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.755389 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.755486 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.755421 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.755486 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.755441 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.755486 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.755479 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.755672 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.755507 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4e0926-157d-4f36-874d-bf2af64dbc79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.755672 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.755569 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856198 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856117 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856198 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856153 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8s2\" (UniqueName: \"kubernetes.io/projected/9c4e0926-157d-4f36-874d-bf2af64dbc79-kube-api-access-lt8s2\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856198 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856171 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856198 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856197 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856497 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856242 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856497 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856259 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856497 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856303 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856497 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856337 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4e0926-157d-4f36-874d-bf2af64dbc79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856497 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856368 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856719 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856552 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856719 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856614 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856786 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856755 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.856786 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.856757 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.857119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.857100 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4e0926-157d-4f36-874d-bf2af64dbc79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.858630 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.858600 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.858767 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.858748 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.863477 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.863450 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4e0926-157d-4f36-874d-bf2af64dbc79-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.863797 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.863781 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8s2\" (UniqueName: \"kubernetes.io/projected/9c4e0926-157d-4f36-874d-bf2af64dbc79-kube-api-access-lt8s2\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8cwk8\" (UID: \"9c4e0926-157d-4f36-874d-bf2af64dbc79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:47.896083 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:47.896061 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:48.027940 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:48.027906 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8"] Aug 04 00:13:48.030263 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:13:48.030236 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4e0926_157d_4f36_874d_bf2af64dbc79.slice/crio-66dab9dd4c36aa46485a26fea3c0b4e1950377df8ad9c700f13d708946e79e39 WatchSource:0}: Error finding container 66dab9dd4c36aa46485a26fea3c0b4e1950377df8ad9c700f13d708946e79e39: Status 404 returned error can't find the container with id 66dab9dd4c36aa46485a26fea3c0b4e1950377df8ad9c700f13d708946e79e39 Aug 04 00:13:48.032402 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:48.032366 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:13:48.032478 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:48.032449 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:13:48.032513 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:48.032489 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:13:48.366681 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:48.366646 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" event={"ID":"9c4e0926-157d-4f36-874d-bf2af64dbc79","Type":"ContainerStarted","Data":"032e2d08f229fd210b37b782c02799c3002eecb1c41c4e1e38fbdd2bffc10c6f"} Aug 04 00:13:48.366907 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:48.366704 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" event={"ID":"9c4e0926-157d-4f36-874d-bf2af64dbc79","Type":"ContainerStarted","Data":"66dab9dd4c36aa46485a26fea3c0b4e1950377df8ad9c700f13d708946e79e39"} Aug 04 00:13:48.388723 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:48.388677 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" podStartSLOduration=1.388662583 podStartE2EDuration="1.388662583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:13:48.388059447 +0000 UTC m=+667.050968927" watchObservedRunningTime="2026-08-04 00:13:48.388662583 +0000 UTC m=+667.051572061" Aug 04 00:13:48.897142 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:48.897105 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:49.901139 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:49.901108 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:50.376224 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:50.376184 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:13:50.377015 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:13:50.376997 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:14:00.481643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:00.481604 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-hklk7"] Aug 04 00:14:00.485209 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:00.485186 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-hklk7" Aug 04 00:14:00.487579 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:00.487558 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zg8cz\"" Aug 04 00:14:00.491037 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:00.491017 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hklk7"] Aug 04 00:14:00.552068 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:00.552040 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ggr\" (UniqueName: \"kubernetes.io/projected/65ef4e00-a13c-49e8-8d13-b1a922947240-kube-api-access-l9ggr\") pod \"authorino-b54df94b4-hklk7\" (UID: \"65ef4e00-a13c-49e8-8d13-b1a922947240\") " pod="kuadrant-system/authorino-b54df94b4-hklk7" Aug 04 00:14:00.653055 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:00.653015 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ggr\" (UniqueName: \"kubernetes.io/projected/65ef4e00-a13c-49e8-8d13-b1a922947240-kube-api-access-l9ggr\") pod \"authorino-b54df94b4-hklk7\" (UID: \"65ef4e00-a13c-49e8-8d13-b1a922947240\") " pod="kuadrant-system/authorino-b54df94b4-hklk7" Aug 04 00:14:00.660936 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:00.660901 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ggr\" (UniqueName: \"kubernetes.io/projected/65ef4e00-a13c-49e8-8d13-b1a922947240-kube-api-access-l9ggr\") pod \"authorino-b54df94b4-hklk7\" (UID: \"65ef4e00-a13c-49e8-8d13-b1a922947240\") " pod="kuadrant-system/authorino-b54df94b4-hklk7" Aug 04 00:14:00.796197 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:00.796111 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-hklk7" Aug 04 00:14:00.924982 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:00.924960 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hklk7"] Aug 04 00:14:00.927212 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:14:00.927184 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65ef4e00_a13c_49e8_8d13_b1a922947240.slice/crio-b0c4e46033b0a248e5eddcdf26be130ec50c8f5daaf90030a2852e10b5ed0d22 WatchSource:0}: Error finding container b0c4e46033b0a248e5eddcdf26be130ec50c8f5daaf90030a2852e10b5ed0d22: Status 404 returned error can't find the container with id b0c4e46033b0a248e5eddcdf26be130ec50c8f5daaf90030a2852e10b5ed0d22 Aug 04 00:14:01.434145 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:01.434113 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-hklk7" event={"ID":"65ef4e00-a13c-49e8-8d13-b1a922947240","Type":"ContainerStarted","Data":"b0c4e46033b0a248e5eddcdf26be130ec50c8f5daaf90030a2852e10b5ed0d22"} Aug 04 00:14:04.453971 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:04.453933 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-hklk7" event={"ID":"65ef4e00-a13c-49e8-8d13-b1a922947240","Type":"ContainerStarted","Data":"a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31"} Aug 04 00:14:04.469493 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:04.469440 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-hklk7" podStartSLOduration=1.882994418 podStartE2EDuration="4.469425138s" totalImagesPullingTime="2.58643072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:14:04.468325176 +0000 UTC m=+683.131234655" watchObservedRunningTime="2026-08-04 00:14:04.469425138 +0000 UTC m=+683.132334616" Aug 04 00:14:06.861743 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:06.861710 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hklk7"] Aug 04 00:14:06.862170 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:06.861885 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-hklk7" podUID="65ef4e00-a13c-49e8-8d13-b1a922947240" containerName="authorino" containerID="cri-o://a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31" gracePeriod=30 Aug 04 00:14:07.109748 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.109725 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-hklk7" Aug 04 00:14:07.211409 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.211324 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9ggr\" (UniqueName: \"kubernetes.io/projected/65ef4e00-a13c-49e8-8d13-b1a922947240-kube-api-access-l9ggr\") pod \"65ef4e00-a13c-49e8-8d13-b1a922947240\" (UID: \"65ef4e00-a13c-49e8-8d13-b1a922947240\") " Aug 04 00:14:07.213372 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.213347 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ef4e00-a13c-49e8-8d13-b1a922947240-kube-api-access-l9ggr" (OuterVolumeSpecName: "kube-api-access-l9ggr") pod "65ef4e00-a13c-49e8-8d13-b1a922947240" (UID: "65ef4e00-a13c-49e8-8d13-b1a922947240"). InnerVolumeSpecName "kube-api-access-l9ggr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:14:07.312220 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.312188 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9ggr\" (UniqueName: \"kubernetes.io/projected/65ef4e00-a13c-49e8-8d13-b1a922947240-kube-api-access-l9ggr\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:14:07.467725 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.467642 2552 generic.go:356] "Generic (PLEG): container finished" podID="65ef4e00-a13c-49e8-8d13-b1a922947240" containerID="a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31" exitCode=0 Aug 04 00:14:07.467725 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.467682 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-hklk7" Aug 04 00:14:07.467909 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.467726 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-hklk7" event={"ID":"65ef4e00-a13c-49e8-8d13-b1a922947240","Type":"ContainerDied","Data":"a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31"} Aug 04 00:14:07.467909 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.467766 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-hklk7" event={"ID":"65ef4e00-a13c-49e8-8d13-b1a922947240","Type":"ContainerDied","Data":"b0c4e46033b0a248e5eddcdf26be130ec50c8f5daaf90030a2852e10b5ed0d22"} Aug 04 00:14:07.467909 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.467787 2552 scope.go:117] "RemoveContainer" containerID="a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31" Aug 04 00:14:07.476438 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.476420 2552 scope.go:117] "RemoveContainer" containerID="a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31" Aug 04 00:14:07.476689 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:14:07.476670 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31\": container with ID starting with a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31 not found: ID does not exist" containerID="a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31" Aug 04 00:14:07.476751 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.476702 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31"} err="failed to get container status \"a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31\": rpc error: code = NotFound desc = could not find container \"a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31\": container with ID starting with a688173f9a9276f9cdcca1dd2a10aed29d32bdb9a594e46ce110b25649030a31 not found: ID does not exist" Aug 04 00:14:07.492366 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.492345 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hklk7"] Aug 04 00:14:07.499268 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.499249 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hklk7"] Aug 04 00:14:07.952175 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:07.952138 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ef4e00-a13c-49e8-8d13-b1a922947240" path="/var/lib/kubelet/pods/65ef4e00-a13c-49e8-8d13-b1a922947240/volumes" Aug 04 00:14:33.778522 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:33.778441 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-5ns7n"] Aug 04 00:14:33.778991 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:33.778953 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ef4e00-a13c-49e8-8d13-b1a922947240" containerName="authorino" Aug 04 00:14:33.778991 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:33.778971 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ef4e00-a13c-49e8-8d13-b1a922947240" containerName="authorino" Aug 04 00:14:33.779112 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:33.779063 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ef4e00-a13c-49e8-8d13-b1a922947240" containerName="authorino" Aug 04 00:14:33.784397 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:33.784377 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-5ns7n" Aug 04 00:14:33.787267 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:33.787241 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zg8cz\"" Aug 04 00:14:33.787928 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:33.787907 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-5ns7n"] Aug 04 00:14:33.928744 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:33.928711 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpqsq\" (UniqueName: \"kubernetes.io/projected/5df001b6-756c-49fd-85db-b10226f62a96-kube-api-access-tpqsq\") pod \"authorino-5dc7ccb5f-5ns7n\" (UID: \"5df001b6-756c-49fd-85db-b10226f62a96\") " pod="kuadrant-system/authorino-5dc7ccb5f-5ns7n" Aug 04 00:14:34.010051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.010015 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-5ns7n"] Aug 04 00:14:34.010257 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:14:34.010237 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tpqsq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-5ns7n" podUID="5df001b6-756c-49fd-85db-b10226f62a96" Aug 04 00:14:34.029359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.029289 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpqsq\" (UniqueName: \"kubernetes.io/projected/5df001b6-756c-49fd-85db-b10226f62a96-kube-api-access-tpqsq\") pod \"authorino-5dc7ccb5f-5ns7n\" (UID: \"5df001b6-756c-49fd-85db-b10226f62a96\") " pod="kuadrant-system/authorino-5dc7ccb5f-5ns7n" Aug 04 00:14:34.029820 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.029802 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c5ff8fc8f-zdfwz"] Aug 04 00:14:34.033727 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.033707 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" Aug 04 00:14:34.037724 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.037705 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpqsq\" (UniqueName: \"kubernetes.io/projected/5df001b6-756c-49fd-85db-b10226f62a96-kube-api-access-tpqsq\") pod \"authorino-5dc7ccb5f-5ns7n\" (UID: \"5df001b6-756c-49fd-85db-b10226f62a96\") " pod="kuadrant-system/authorino-5dc7ccb5f-5ns7n" Aug 04 00:14:34.044276 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.044243 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c5ff8fc8f-zdfwz"] Aug 04 00:14:34.130194 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.130155 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlpvp\" (UniqueName: \"kubernetes.io/projected/16862a82-bf64-449e-980f-e6d09307759f-kube-api-access-nlpvp\") pod \"authorino-5c5ff8fc8f-zdfwz\" (UID: \"16862a82-bf64-449e-980f-e6d09307759f\") " pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" Aug 04 00:14:34.230720 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.230688 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlpvp\" (UniqueName: \"kubernetes.io/projected/16862a82-bf64-449e-980f-e6d09307759f-kube-api-access-nlpvp\") pod \"authorino-5c5ff8fc8f-zdfwz\" (UID: \"16862a82-bf64-449e-980f-e6d09307759f\") " pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" Aug 04 00:14:34.239169 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.239139 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlpvp\" (UniqueName: \"kubernetes.io/projected/16862a82-bf64-449e-980f-e6d09307759f-kube-api-access-nlpvp\") pod \"authorino-5c5ff8fc8f-zdfwz\" (UID: \"16862a82-bf64-449e-980f-e6d09307759f\") " pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" Aug 04 00:14:34.351880 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.351785 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" Aug 04 00:14:34.478893 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.478842 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c5ff8fc8f-zdfwz"] Aug 04 00:14:34.482228 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:14:34.482202 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16862a82_bf64_449e_980f_e6d09307759f.slice/crio-5aa25c58adcb4f0c44347fb90f97b03678b92cce4b3bd1866718087396dd2c8e WatchSource:0}: Error finding container 5aa25c58adcb4f0c44347fb90f97b03678b92cce4b3bd1866718087396dd2c8e: Status 404 returned error can't find the container with id 5aa25c58adcb4f0c44347fb90f97b03678b92cce4b3bd1866718087396dd2c8e Aug 04 00:14:34.588636 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.588602 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" event={"ID":"16862a82-bf64-449e-980f-e6d09307759f","Type":"ContainerStarted","Data":"5aa25c58adcb4f0c44347fb90f97b03678b92cce4b3bd1866718087396dd2c8e"} Aug 04 00:14:34.588636 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.588620 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-5ns7n" Aug 04 00:14:34.596627 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.596608 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-5ns7n" Aug 04 00:14:34.735260 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.735231 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpqsq\" (UniqueName: \"kubernetes.io/projected/5df001b6-756c-49fd-85db-b10226f62a96-kube-api-access-tpqsq\") pod \"5df001b6-756c-49fd-85db-b10226f62a96\" (UID: \"5df001b6-756c-49fd-85db-b10226f62a96\") " Aug 04 00:14:34.737277 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.737256 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df001b6-756c-49fd-85db-b10226f62a96-kube-api-access-tpqsq" (OuterVolumeSpecName: "kube-api-access-tpqsq") pod "5df001b6-756c-49fd-85db-b10226f62a96" (UID: "5df001b6-756c-49fd-85db-b10226f62a96"). InnerVolumeSpecName "kube-api-access-tpqsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:14:34.836734 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:34.836703 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpqsq\" (UniqueName: \"kubernetes.io/projected/5df001b6-756c-49fd-85db-b10226f62a96-kube-api-access-tpqsq\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:14:35.594082 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.594004 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" event={"ID":"16862a82-bf64-449e-980f-e6d09307759f","Type":"ContainerStarted","Data":"e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5"} Aug 04 00:14:35.594082 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.594017 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-5ns7n" Aug 04 00:14:35.610537 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.610486 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" podStartSLOduration=0.849443083 podStartE2EDuration="1.610467916s" totalImagesPullingTime="761.024833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:14:35.610113771 +0000 UTC m=+714.273023249" watchObservedRunningTime="2026-08-04 00:14:35.610467916 +0000 UTC m=+714.273377395" Aug 04 00:14:35.634157 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.634127 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-5ns7n"] Aug 04 00:14:35.643983 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.643954 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-5ns7n"] Aug 04 00:14:35.739683 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.739649 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cb5865d79-567n6"] Aug 04 00:14:35.742707 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.742678 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:14:35.745920 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.745894 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 00:14:35.758753 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.758727 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cb5865d79-567n6"] Aug 04 00:14:35.844458 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.844377 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbl6v\" (UniqueName: \"kubernetes.io/projected/a3a52260-5b84-4681-9499-021aedf7dbae-kube-api-access-cbl6v\") pod \"authorino-5cb5865d79-567n6\" (UID: \"a3a52260-5b84-4681-9499-021aedf7dbae\") " pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:14:35.844458 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.844431 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3a52260-5b84-4681-9499-021aedf7dbae-tls-cert\") pod \"authorino-5cb5865d79-567n6\" (UID: \"a3a52260-5b84-4681-9499-021aedf7dbae\") " pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:14:35.945814 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.945775 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbl6v\" (UniqueName: \"kubernetes.io/projected/a3a52260-5b84-4681-9499-021aedf7dbae-kube-api-access-cbl6v\") pod \"authorino-5cb5865d79-567n6\" (UID: \"a3a52260-5b84-4681-9499-021aedf7dbae\") " pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:14:35.946016 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.945843 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3a52260-5b84-4681-9499-021aedf7dbae-tls-cert\") pod \"authorino-5cb5865d79-567n6\" (UID: \"a3a52260-5b84-4681-9499-021aedf7dbae\") " pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:14:35.948418 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.948387 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3a52260-5b84-4681-9499-021aedf7dbae-tls-cert\") pod \"authorino-5cb5865d79-567n6\" (UID: \"a3a52260-5b84-4681-9499-021aedf7dbae\") " pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:14:35.954082 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.954052 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5df001b6-756c-49fd-85db-b10226f62a96" path="/var/lib/kubelet/pods/5df001b6-756c-49fd-85db-b10226f62a96/volumes" Aug 04 00:14:35.956547 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:35.956525 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbl6v\" (UniqueName: \"kubernetes.io/projected/a3a52260-5b84-4681-9499-021aedf7dbae-kube-api-access-cbl6v\") pod \"authorino-5cb5865d79-567n6\" (UID: \"a3a52260-5b84-4681-9499-021aedf7dbae\") " pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:14:36.052735 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:36.052699 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:14:36.182672 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:36.182649 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cb5865d79-567n6"] Aug 04 00:14:36.184746 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:14:36.184716 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3a52260_5b84_4681_9499_021aedf7dbae.slice/crio-2c1a5bedb0ff3833ee470cce0471fc9832a074a413413997b2f4a5cc871957d9 WatchSource:0}: Error finding container 2c1a5bedb0ff3833ee470cce0471fc9832a074a413413997b2f4a5cc871957d9: Status 404 returned error can't find the container with id 2c1a5bedb0ff3833ee470cce0471fc9832a074a413413997b2f4a5cc871957d9 Aug 04 00:14:36.600559 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:36.600529 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cb5865d79-567n6" event={"ID":"a3a52260-5b84-4681-9499-021aedf7dbae","Type":"ContainerStarted","Data":"2c1a5bedb0ff3833ee470cce0471fc9832a074a413413997b2f4a5cc871957d9"} Aug 04 00:14:37.608292 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:37.608255 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cb5865d79-567n6" event={"ID":"a3a52260-5b84-4681-9499-021aedf7dbae","Type":"ContainerStarted","Data":"24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01"} Aug 04 00:14:37.625383 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:37.625332 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5cb5865d79-567n6" podStartSLOduration=2.131877998 podStartE2EDuration="2.625315366s" totalImagesPullingTime="493.437368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:14:37.623669547 +0000 UTC m=+716.286579026" watchObservedRunningTime="2026-08-04 00:14:37.625315366 +0000 UTC m=+716.288224891" Aug 04 00:14:37.652815 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:37.652769 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c5ff8fc8f-zdfwz"] Aug 04 00:14:37.653429 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:37.653398 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" podUID="16862a82-bf64-449e-980f-e6d09307759f" containerName="authorino" containerID="cri-o://e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5" gracePeriod=30 Aug 04 00:14:37.903880 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:37.903836 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" Aug 04 00:14:37.963111 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:37.961298 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlpvp\" (UniqueName: \"kubernetes.io/projected/16862a82-bf64-449e-980f-e6d09307759f-kube-api-access-nlpvp\") pod \"16862a82-bf64-449e-980f-e6d09307759f\" (UID: \"16862a82-bf64-449e-980f-e6d09307759f\") " Aug 04 00:14:37.966840 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:37.966807 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16862a82-bf64-449e-980f-e6d09307759f-kube-api-access-nlpvp" (OuterVolumeSpecName: "kube-api-access-nlpvp") pod "16862a82-bf64-449e-980f-e6d09307759f" (UID: "16862a82-bf64-449e-980f-e6d09307759f"). InnerVolumeSpecName "kube-api-access-nlpvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:14:38.062549 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.062512 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlpvp\" (UniqueName: \"kubernetes.io/projected/16862a82-bf64-449e-980f-e6d09307759f-kube-api-access-nlpvp\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:14:38.614679 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.614648 2552 generic.go:356] "Generic (PLEG): container finished" podID="16862a82-bf64-449e-980f-e6d09307759f" containerID="e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5" exitCode=0 Aug 04 00:14:38.615103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.614705 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" Aug 04 00:14:38.615103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.614739 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" event={"ID":"16862a82-bf64-449e-980f-e6d09307759f","Type":"ContainerDied","Data":"e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5"} Aug 04 00:14:38.615103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.614782 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c5ff8fc8f-zdfwz" event={"ID":"16862a82-bf64-449e-980f-e6d09307759f","Type":"ContainerDied","Data":"5aa25c58adcb4f0c44347fb90f97b03678b92cce4b3bd1866718087396dd2c8e"} Aug 04 00:14:38.615103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.614804 2552 scope.go:117] "RemoveContainer" containerID="e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5" Aug 04 00:14:38.623497 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.623482 2552 scope.go:117] "RemoveContainer" containerID="e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5" Aug 04 00:14:38.623748 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:14:38.623729 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5\": container with ID starting with e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5 not found: ID does not exist" containerID="e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5" Aug 04 00:14:38.623799 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.623756 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5"} err="failed to get container status \"e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5\": rpc error: code = NotFound desc = could not find container \"e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5\": container with ID starting with e8e653c9c89fab72f1195d4bb0c321056bd214e5bbe09ca8c9c0b6974d5176d5 not found: ID does not exist" Aug 04 00:14:38.634342 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.634313 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c5ff8fc8f-zdfwz"] Aug 04 00:14:38.643088 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:38.643068 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c5ff8fc8f-zdfwz"] Aug 04 00:14:39.953447 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:14:39.953416 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16862a82-bf64-449e-980f-e6d09307759f" path="/var/lib/kubelet/pods/16862a82-bf64-449e-980f-e6d09307759f/volumes" Aug 04 00:15:17.323836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.323803 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-j2q5h"] Aug 04 00:15:17.324319 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.324173 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16862a82-bf64-449e-980f-e6d09307759f" containerName="authorino" Aug 04 00:15:17.324319 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.324184 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="16862a82-bf64-449e-980f-e6d09307759f" containerName="authorino" Aug 04 00:15:17.324319 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.324245 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="16862a82-bf64-449e-980f-e6d09307759f" containerName="authorino" Aug 04 00:15:17.327398 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.327371 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:17.329716 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.329686 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Aug 04 00:15:17.329997 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.329982 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gn6px\"" Aug 04 00:15:17.333708 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.333686 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-j2q5h"] Aug 04 00:15:17.400739 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.400700 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2tj\" (UniqueName: \"kubernetes.io/projected/d1436586-9a7f-4366-a7d1-c504cbac172e-kube-api-access-gs2tj\") pod \"payload-processing-5cc6cd444b-j2q5h\" (UID: \"d1436586-9a7f-4366-a7d1-c504cbac172e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:17.400920 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.400759 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1436586-9a7f-4366-a7d1-c504cbac172e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-j2q5h\" (UID: \"d1436586-9a7f-4366-a7d1-c504cbac172e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:17.501947 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.501909 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2tj\" (UniqueName: \"kubernetes.io/projected/d1436586-9a7f-4366-a7d1-c504cbac172e-kube-api-access-gs2tj\") pod \"payload-processing-5cc6cd444b-j2q5h\" (UID: \"d1436586-9a7f-4366-a7d1-c504cbac172e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:17.502135 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.501973 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1436586-9a7f-4366-a7d1-c504cbac172e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-j2q5h\" (UID: \"d1436586-9a7f-4366-a7d1-c504cbac172e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:17.502955 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.502925 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1436586-9a7f-4366-a7d1-c504cbac172e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-j2q5h\" (UID: \"d1436586-9a7f-4366-a7d1-c504cbac172e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:17.511360 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.511335 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2tj\" (UniqueName: \"kubernetes.io/projected/d1436586-9a7f-4366-a7d1-c504cbac172e-kube-api-access-gs2tj\") pod \"payload-processing-5cc6cd444b-j2q5h\" (UID: \"d1436586-9a7f-4366-a7d1-c504cbac172e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:17.640112 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.640082 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:17.772036 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.772014 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-j2q5h"] Aug 04 00:15:17.773928 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:15:17.773893 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1436586_9a7f_4366_a7d1_c504cbac172e.slice/crio-aa8c863e94ea9d7083aeae0d7ec2413b6d32002fa71a7d7a9a4d9c0c065ed42c WatchSource:0}: Error finding container aa8c863e94ea9d7083aeae0d7ec2413b6d32002fa71a7d7a9a4d9c0c065ed42c: Status 404 returned error can't find the container with id aa8c863e94ea9d7083aeae0d7ec2413b6d32002fa71a7d7a9a4d9c0c065ed42c Aug 04 00:15:17.790782 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:17.790745 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" event={"ID":"d1436586-9a7f-4366-a7d1-c504cbac172e","Type":"ContainerStarted","Data":"aa8c863e94ea9d7083aeae0d7ec2413b6d32002fa71a7d7a9a4d9c0c065ed42c"} Aug 04 00:15:21.817672 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:21.817640 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" event={"ID":"d1436586-9a7f-4366-a7d1-c504cbac172e","Type":"ContainerStarted","Data":"968991f140aca565f6dfe000a6f7904944abadb9de0aac8d1b32a052589f5fca"} Aug 04 00:15:21.818100 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:21.817734 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:21.833830 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:21.833782 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" podStartSLOduration=1.819673399 podStartE2EDuration="4.833766241s" totalImagesPullingTime="3.014092842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:21.832346977 +0000 UTC m=+760.495256455" watchObservedRunningTime="2026-08-04 00:15:21.833766241 +0000 UTC m=+760.496675719" Aug 04 00:15:26.842334 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:26.842230 2552 generic.go:356] "Generic (PLEG): container finished" podID="9c4e0926-157d-4f36-874d-bf2af64dbc79" containerID="032e2d08f229fd210b37b782c02799c3002eecb1c41c4e1e38fbdd2bffc10c6f" exitCode=0 Aug 04 00:15:26.842334 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:26.842302 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" event={"ID":"9c4e0926-157d-4f36-874d-bf2af64dbc79","Type":"ContainerDied","Data":"032e2d08f229fd210b37b782c02799c3002eecb1c41c4e1e38fbdd2bffc10c6f"} Aug 04 00:15:26.842740 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:26.842667 2552 scope.go:117] "RemoveContainer" containerID="032e2d08f229fd210b37b782c02799c3002eecb1c41c4e1e38fbdd2bffc10c6f" Aug 04 00:15:26.843651 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:26.843625 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:15:26.843721 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:26.843705 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:15:26.843769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:26.843754 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:15:27.848748 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:27.848715 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" event={"ID":"9c4e0926-157d-4f36-874d-bf2af64dbc79","Type":"ContainerStarted","Data":"fc92d905f6a3db67c3b6809724994fa966ff1ef2c433ab642646ca90b8bbe069"} Aug 04 00:15:27.896209 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:27.896174 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:15:27.896520 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:27.896494 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8cwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Aug 04 00:15:27.896635 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:27.896548 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" podUID="9c4e0926-157d-4f36-874d-bf2af64dbc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Aug 04 00:15:28.897881 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:28.897179 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8cwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Aug 04 00:15:28.897881 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:28.897242 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" podUID="9c4e0926-157d-4f36-874d-bf2af64dbc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Aug 04 00:15:29.896559 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:29.896519 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8cwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Aug 04 00:15:29.896734 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:29.896596 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" podUID="9c4e0926-157d-4f36-874d-bf2af64dbc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Aug 04 00:15:30.897074 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:30.897029 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8cwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Aug 04 00:15:30.897457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:30.897099 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" podUID="9c4e0926-157d-4f36-874d-bf2af64dbc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Aug 04 00:15:31.897044 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:31.896997 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8cwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Aug 04 00:15:31.897260 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:31.897061 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" podUID="9c4e0926-157d-4f36-874d-bf2af64dbc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Aug 04 00:15:32.823952 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:32.823927 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-j2q5h" Aug 04 00:15:32.897367 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:32.897336 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8cwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Aug 04 00:15:32.897716 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:32.897392 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" podUID="9c4e0926-157d-4f36-874d-bf2af64dbc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Aug 04 00:15:33.914548 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:33.914506 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:15:33.914979 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:33.914891 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:15:33.915778 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:15:33.915759 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8cwk8" Aug 04 00:16:18.927614 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:18.927527 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2"] Aug 04 00:16:18.931232 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:18.931210 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:18.934457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:18.934434 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k7knc\"" Aug 04 00:16:18.934457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:18.934449 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 04 00:16:18.934628 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:18.934470 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 00:16:18.934628 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:18.934477 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 00:16:18.940590 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:18.940568 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2"] Aug 04 00:16:19.001639 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.001605 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f4feea-d897-47ae-81d1-bec12aae84d2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.001850 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.001661 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.001850 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.001710 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.001850 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.001744 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.001850 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.001801 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk248\" (UniqueName: \"kubernetes.io/projected/52f4feea-d897-47ae-81d1-bec12aae84d2-kube-api-access-bk248\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.002039 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.001953 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.102929 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.102901 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk248\" (UniqueName: \"kubernetes.io/projected/52f4feea-d897-47ae-81d1-bec12aae84d2-kube-api-access-bk248\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.103103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.102962 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.103103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.102997 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f4feea-d897-47ae-81d1-bec12aae84d2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.103103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.103037 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.103103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.103064 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.103292 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.103200 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.103458 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.103438 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.103494 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.103458 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.103531 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.103499 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.105381 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.105345 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f4feea-d897-47ae-81d1-bec12aae84d2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.105644 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.105627 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f4feea-d897-47ae-81d1-bec12aae84d2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.110323 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.110302 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk248\" (UniqueName: \"kubernetes.io/projected/52f4feea-d897-47ae-81d1-bec12aae84d2-kube-api-access-bk248\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qpcg2\" (UID: \"52f4feea-d897-47ae-81d1-bec12aae84d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.242236 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.242157 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:19.377255 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.377123 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2"] Aug 04 00:16:19.379852 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:16:19.379827 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f4feea_d897_47ae_81d1_bec12aae84d2.slice/crio-f45d151dbf44988345bde43bcf03d3376661c3de9bd26ec4f2f358ebc515cfe8 WatchSource:0}: Error finding container f45d151dbf44988345bde43bcf03d3376661c3de9bd26ec4f2f358ebc515cfe8: Status 404 returned error can't find the container with id f45d151dbf44988345bde43bcf03d3376661c3de9bd26ec4f2f358ebc515cfe8 Aug 04 00:16:19.381275 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:19.381255 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:16:20.089459 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:20.089421 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" event={"ID":"52f4feea-d897-47ae-81d1-bec12aae84d2","Type":"ContainerStarted","Data":"f45d151dbf44988345bde43bcf03d3376661c3de9bd26ec4f2f358ebc515cfe8"} Aug 04 00:16:21.096018 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:21.095985 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" event={"ID":"52f4feea-d897-47ae-81d1-bec12aae84d2","Type":"ContainerStarted","Data":"7a385620fd4b2fda9db8e6cc7df2e5a23eb15178bd42198f6641d921d3b4b7c1"} Aug 04 00:16:21.113957 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:21.113911 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" podStartSLOduration=1.574389091 podStartE2EDuration="3.113894688s" totalImagesPullingTime="1.539505597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:16:21.111551045 +0000 UTC m=+819.774460525" watchObservedRunningTime="2026-08-04 00:16:21.113894688 +0000 UTC m=+819.776804166" Aug 04 00:16:29.243099 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:29.243058 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:29.243491 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:29.243123 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:29.255655 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:29.255629 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:30.150509 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:30.150479 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qpcg2" Aug 04 00:16:35.126177 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.125903 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx"] Aug 04 00:16:35.129543 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.129521 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.131712 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.131692 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 04 00:16:35.137260 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.137238 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx"] Aug 04 00:16:35.249912 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.249879 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f2dr\" (UniqueName: \"kubernetes.io/projected/1beb34df-a3db-40f6-b76a-1e4d29793d0b-kube-api-access-7f2dr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.250132 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.249926 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.250132 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.249945 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.250132 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.250021 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.250132 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.250043 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.250132 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.250062 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1beb34df-a3db-40f6-b76a-1e4d29793d0b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.350489 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.350452 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.350489 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.350491 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.350699 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.350578 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.350699 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.350616 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.350699 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.350645 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1beb34df-a3db-40f6-b76a-1e4d29793d0b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.350699 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.350681 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f2dr\" (UniqueName: \"kubernetes.io/projected/1beb34df-a3db-40f6-b76a-1e4d29793d0b-kube-api-access-7f2dr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.350975 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.350952 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.351096 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.351063 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.351212 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.351094 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.352813 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.352784 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1beb34df-a3db-40f6-b76a-1e4d29793d0b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.353112 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.353091 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1beb34df-a3db-40f6-b76a-1e4d29793d0b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.358988 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.358968 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f2dr\" (UniqueName: \"kubernetes.io/projected/1beb34df-a3db-40f6-b76a-1e4d29793d0b-kube-api-access-7f2dr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx\" (UID: \"1beb34df-a3db-40f6-b76a-1e4d29793d0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.441271 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.441185 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:35.780202 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:35.774119 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx"] Aug 04 00:16:36.168243 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:36.168212 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" event={"ID":"1beb34df-a3db-40f6-b76a-1e4d29793d0b","Type":"ContainerStarted","Data":"660da9c3e2a042148a21b487319e51aaf67502409fce9a94897f9c71e7af93eb"} Aug 04 00:16:36.168243 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:36.168250 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" event={"ID":"1beb34df-a3db-40f6-b76a-1e4d29793d0b","Type":"ContainerStarted","Data":"3868351af9d6c3e33cf76d3aef8fcbb1e05fab7f88b0845a97b3847dc2f9c0bf"} Aug 04 00:16:36.187782 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:36.187728 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" podStartSLOduration=1.00186183 podStartE2EDuration="1.187710435s" totalImagesPullingTime="185.848605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:16:36.185541993 +0000 UTC m=+834.848451471" watchObservedRunningTime="2026-08-04 00:16:36.187710435 +0000 UTC m=+834.850619913" Aug 04 00:16:45.441942 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:45.441904 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:45.442343 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:45.441960 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:45.454623 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:45.454595 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:16:46.229216 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:16:46.229183 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx" Aug 04 00:18:04.029357 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.029322 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cb5865d79-567n6"] Aug 04 00:18:04.030079 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.029515 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5cb5865d79-567n6" podUID="a3a52260-5b84-4681-9499-021aedf7dbae" containerName="authorino" containerID="cri-o://24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01" gracePeriod=30 Aug 04 00:18:04.275553 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.275528 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:18:04.328966 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.328897 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3a52260-5b84-4681-9499-021aedf7dbae-tls-cert\") pod \"a3a52260-5b84-4681-9499-021aedf7dbae\" (UID: \"a3a52260-5b84-4681-9499-021aedf7dbae\") " Aug 04 00:18:04.328966 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.328947 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbl6v\" (UniqueName: \"kubernetes.io/projected/a3a52260-5b84-4681-9499-021aedf7dbae-kube-api-access-cbl6v\") pod \"a3a52260-5b84-4681-9499-021aedf7dbae\" (UID: \"a3a52260-5b84-4681-9499-021aedf7dbae\") " Aug 04 00:18:04.330906 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.330872 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a52260-5b84-4681-9499-021aedf7dbae-kube-api-access-cbl6v" (OuterVolumeSpecName: "kube-api-access-cbl6v") pod "a3a52260-5b84-4681-9499-021aedf7dbae" (UID: "a3a52260-5b84-4681-9499-021aedf7dbae"). InnerVolumeSpecName "kube-api-access-cbl6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:18:04.338508 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.338479 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a52260-5b84-4681-9499-021aedf7dbae-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a3a52260-5b84-4681-9499-021aedf7dbae" (UID: "a3a52260-5b84-4681-9499-021aedf7dbae"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:18:04.430309 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.430266 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbl6v\" (UniqueName: \"kubernetes.io/projected/a3a52260-5b84-4681-9499-021aedf7dbae-kube-api-access-cbl6v\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:18:04.430309 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.430304 2552 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3a52260-5b84-4681-9499-021aedf7dbae-tls-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:18:04.578426 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.578392 2552 generic.go:356] "Generic (PLEG): container finished" podID="a3a52260-5b84-4681-9499-021aedf7dbae" containerID="24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01" exitCode=0 Aug 04 00:18:04.578614 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.578436 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cb5865d79-567n6" Aug 04 00:18:04.578614 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.578481 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cb5865d79-567n6" event={"ID":"a3a52260-5b84-4681-9499-021aedf7dbae","Type":"ContainerDied","Data":"24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01"} Aug 04 00:18:04.578614 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.578524 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cb5865d79-567n6" event={"ID":"a3a52260-5b84-4681-9499-021aedf7dbae","Type":"ContainerDied","Data":"2c1a5bedb0ff3833ee470cce0471fc9832a074a413413997b2f4a5cc871957d9"} Aug 04 00:18:04.578614 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.578545 2552 scope.go:117] "RemoveContainer" containerID="24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01" Aug 04 00:18:04.587251 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.587234 2552 scope.go:117] "RemoveContainer" containerID="24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01" Aug 04 00:18:04.587474 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:18:04.587457 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01\": container with ID starting with 24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01 not found: ID does not exist" containerID="24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01" Aug 04 00:18:04.587542 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.587487 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01"} err="failed to get container status \"24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01\": rpc error: code = NotFound desc = could not find container \"24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01\": container with ID starting with 24b171bed9ce9abd6b8521a5359e85fa9c08c1e8c8ad519845976e694cbacb01 not found: ID does not exist" Aug 04 00:18:04.598979 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.598953 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cb5865d79-567n6"] Aug 04 00:18:04.606072 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:04.606051 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5cb5865d79-567n6"] Aug 04 00:18:05.952964 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:18:05.952932 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3a52260-5b84-4681-9499-021aedf7dbae" path="/var/lib/kubelet/pods/a3a52260-5b84-4681-9499-021aedf7dbae/volumes" Aug 04 00:26:40.768071 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:40.767989 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78"] Aug 04 00:26:40.770528 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:40.768198 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" podUID="a28baae9-478a-476d-b8d3-b1e3c876ef44" containerName="manager" containerID="cri-o://1160bc437f3b9a70dd4fb88eea10febe750961b0da4f28e6dc1268042669b796" gracePeriod=10 Aug 04 00:26:40.962509 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:40.962477 2552 generic.go:356] "Generic (PLEG): container finished" podID="a28baae9-478a-476d-b8d3-b1e3c876ef44" containerID="1160bc437f3b9a70dd4fb88eea10febe750961b0da4f28e6dc1268042669b796" exitCode=0 Aug 04 00:26:40.962655 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:40.962536 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" event={"ID":"a28baae9-478a-476d-b8d3-b1e3c876ef44","Type":"ContainerDied","Data":"1160bc437f3b9a70dd4fb88eea10febe750961b0da4f28e6dc1268042669b796"} Aug 04 00:26:41.032615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.032592 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:26:41.176196 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.176163 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28baae9-478a-476d-b8d3-b1e3c876ef44-extensions-socket-volume\") pod \"a28baae9-478a-476d-b8d3-b1e3c876ef44\" (UID: \"a28baae9-478a-476d-b8d3-b1e3c876ef44\") " Aug 04 00:26:41.176363 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.176278 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5t6m\" (UniqueName: \"kubernetes.io/projected/a28baae9-478a-476d-b8d3-b1e3c876ef44-kube-api-access-w5t6m\") pod \"a28baae9-478a-476d-b8d3-b1e3c876ef44\" (UID: \"a28baae9-478a-476d-b8d3-b1e3c876ef44\") " Aug 04 00:26:41.176554 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.176530 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28baae9-478a-476d-b8d3-b1e3c876ef44-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a28baae9-478a-476d-b8d3-b1e3c876ef44" (UID: "a28baae9-478a-476d-b8d3-b1e3c876ef44"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:26:41.178370 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.178341 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28baae9-478a-476d-b8d3-b1e3c876ef44-kube-api-access-w5t6m" (OuterVolumeSpecName: "kube-api-access-w5t6m") pod "a28baae9-478a-476d-b8d3-b1e3c876ef44" (UID: "a28baae9-478a-476d-b8d3-b1e3c876ef44"). InnerVolumeSpecName "kube-api-access-w5t6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:26:41.277314 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.277237 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5t6m\" (UniqueName: \"kubernetes.io/projected/a28baae9-478a-476d-b8d3-b1e3c876ef44-kube-api-access-w5t6m\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:26:41.277314 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.277265 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28baae9-478a-476d-b8d3-b1e3c876ef44-extensions-socket-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:26:41.931821 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.931793 2552 scope.go:117] "RemoveContainer" containerID="1160bc437f3b9a70dd4fb88eea10febe750961b0da4f28e6dc1268042669b796" Aug 04 00:26:41.966691 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.966657 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" event={"ID":"a28baae9-478a-476d-b8d3-b1e3c876ef44","Type":"ContainerDied","Data":"0fbb529197795eedd2d32c73408bd3e700fa0cceadc281e5f3be1a77877b492f"} Aug 04 00:26:41.966781 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.966678 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78" Aug 04 00:26:41.998371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:41.998345 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78"] Aug 04 00:26:42.001850 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:42.001826 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wzp78"] Aug 04 00:26:43.953256 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:26:43.953226 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a28baae9-478a-476d-b8d3-b1e3c876ef44" path="/var/lib/kubelet/pods/a28baae9-478a-476d-b8d3-b1e3c876ef44/volumes" Aug 04 00:27:41.454786 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.454746 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs"] Aug 04 00:27:41.455405 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.455277 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28baae9-478a-476d-b8d3-b1e3c876ef44" containerName="manager" Aug 04 00:27:41.455405 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.455298 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28baae9-478a-476d-b8d3-b1e3c876ef44" containerName="manager" Aug 04 00:27:41.455405 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.455325 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a52260-5b84-4681-9499-021aedf7dbae" containerName="authorino" Aug 04 00:27:41.455405 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.455333 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a52260-5b84-4681-9499-021aedf7dbae" containerName="authorino" Aug 04 00:27:41.455603 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.455428 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a52260-5b84-4681-9499-021aedf7dbae" containerName="authorino" Aug 04 00:27:41.455603 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.455445 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28baae9-478a-476d-b8d3-b1e3c876ef44" containerName="manager" Aug 04 00:27:41.458298 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.458278 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:27:41.460922 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.460897 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tmg2q\"" Aug 04 00:27:41.474896 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.474875 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs"] Aug 04 00:27:41.496688 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.496665 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-qnmxs\" (UID: \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:27:41.496778 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.496701 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xhr\" (UniqueName: \"kubernetes.io/projected/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-kube-api-access-q8xhr\") pod \"kuadrant-operator-controller-manager-68cf589589-qnmxs\" (UID: \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:27:41.597439 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.597406 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-qnmxs\" (UID: \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:27:41.597589 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.597451 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xhr\" (UniqueName: \"kubernetes.io/projected/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-kube-api-access-q8xhr\") pod \"kuadrant-operator-controller-manager-68cf589589-qnmxs\" (UID: \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:27:41.597788 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.597768 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-qnmxs\" (UID: \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:27:41.612063 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.612040 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xhr\" (UniqueName: \"kubernetes.io/projected/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-kube-api-access-q8xhr\") pod \"kuadrant-operator-controller-manager-68cf589589-qnmxs\" (UID: \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:27:41.769900 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.769805 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tmg2q\"" Aug 04 00:27:41.777606 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.777578 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:27:41.920274 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:41.920249 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs"] Aug 04 00:27:41.922637 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:27:41.922612 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d75ae1_f4bf_459c_ace9_cf9675ddb12c.slice/crio-e930f0c333b0712afa9a83a1f536688fe9e9ada0d1a7c7ac62244098b6f4e1ab WatchSource:0}: Error finding container e930f0c333b0712afa9a83a1f536688fe9e9ada0d1a7c7ac62244098b6f4e1ab: Status 404 returned error can't find the container with id e930f0c333b0712afa9a83a1f536688fe9e9ada0d1a7c7ac62244098b6f4e1ab Aug 04 00:27:42.238217 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:42.238184 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" event={"ID":"87d75ae1-f4bf-459c-ace9-cf9675ddb12c","Type":"ContainerStarted","Data":"f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3"} Aug 04 00:27:42.238441 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:42.238239 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" event={"ID":"87d75ae1-f4bf-459c-ace9-cf9675ddb12c","Type":"ContainerStarted","Data":"e930f0c333b0712afa9a83a1f536688fe9e9ada0d1a7c7ac62244098b6f4e1ab"} Aug 04 00:27:42.238441 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:42.238359 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:27:42.257713 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:42.257667 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" podStartSLOduration=1.25765178 podStartE2EDuration="1.25765178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:27:42.256443491 +0000 UTC m=+1500.919352968" watchObservedRunningTime="2026-08-04 00:27:42.25765178 +0000 UTC m=+1500.920561257" Aug 04 00:27:53.245319 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:27:53.245286 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:30:00.135104 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:00.135012 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd"] Aug 04 00:30:00.138343 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:00.138326 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd" Aug 04 00:30:00.140836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:00.140816 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:30:00.147711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:00.147694 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd" Aug 04 00:30:00.174745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:00.174702 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd"] Aug 04 00:30:00.294323 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:00.294302 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd"] Aug 04 00:30:00.295971 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:30:00.295944 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ca444d_9971_463c_8dab_b3332d6d5473.slice/crio-a3313cbf9e1f833017c7a3965fdeeb1cfe96d3c20156d4df05bc131869c08e05 WatchSource:0}: Error finding container a3313cbf9e1f833017c7a3965fdeeb1cfe96d3c20156d4df05bc131869c08e05: Status 404 returned error can't find the container with id a3313cbf9e1f833017c7a3965fdeeb1cfe96d3c20156d4df05bc131869c08e05 Aug 04 00:30:00.297791 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:00.297774 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:30:00.900691 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:00.900654 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd" event={"ID":"31ca444d-9971-463c-8dab-b3332d6d5473","Type":"ContainerStarted","Data":"a3313cbf9e1f833017c7a3965fdeeb1cfe96d3c20156d4df05bc131869c08e05"} Aug 04 00:30:03.919517 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:03.919484 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd" event={"ID":"31ca444d-9971-463c-8dab-b3332d6d5473","Type":"ContainerStarted","Data":"2ce6917c81da3a3f11bca0a6c5977cad21ba2017b922057592ce2499440f7cc6"} Aug 04 00:30:03.938636 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:03.938587 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd" podStartSLOduration=1.013851501 podStartE2EDuration="3.93857109s" totalImagesPullingTime="2.924719589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:30:03.93666134 +0000 UTC m=+1642.599570819" watchObservedRunningTime="2026-08-04 00:30:03.93857109 +0000 UTC m=+1642.601480568" Aug 04 00:30:04.925122 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:04.925072 2552 generic.go:356] "Generic (PLEG): container finished" podID="31ca444d-9971-463c-8dab-b3332d6d5473" containerID="2ce6917c81da3a3f11bca0a6c5977cad21ba2017b922057592ce2499440f7cc6" exitCode=0 Aug 04 00:30:04.925497 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:04.925152 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd" event={"ID":"31ca444d-9971-463c-8dab-b3332d6d5473","Type":"ContainerDied","Data":"2ce6917c81da3a3f11bca0a6c5977cad21ba2017b922057592ce2499440f7cc6"} Aug 04 00:30:06.081224 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:06.081196 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd" Aug 04 00:30:06.936108 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:06.936083 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd" Aug 04 00:30:06.936282 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:06.936116 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-fbbxd" event={"ID":"31ca444d-9971-463c-8dab-b3332d6d5473","Type":"ContainerDied","Data":"a3313cbf9e1f833017c7a3965fdeeb1cfe96d3c20156d4df05bc131869c08e05"} Aug 04 00:30:06.936282 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:06.936163 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3313cbf9e1f833017c7a3965fdeeb1cfe96d3c20156d4df05bc131869c08e05" Aug 04 00:30:30.833626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.833594 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8"] Aug 04 00:30:30.834033 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.833978 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca444d-9971-463c-8dab-b3332d6d5473" containerName="cleanup" Aug 04 00:30:30.834033 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.833992 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca444d-9971-463c-8dab-b3332d6d5473" containerName="cleanup" Aug 04 00:30:30.834114 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.834065 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ca444d-9971-463c-8dab-b3332d6d5473" containerName="cleanup" Aug 04 00:30:30.837352 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.837330 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:30.840644 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.840626 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3e1ba55f-kserve-self-signed-certs\"" Aug 04 00:30:30.847830 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.847809 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8"] Aug 04 00:30:30.960726 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.960696 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-model-cache\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:30.960899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.960740 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-dshm\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:30.960899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.960761 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tmp-dir\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:30.960899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.960827 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksrc\" (UniqueName: \"kubernetes.io/projected/06d04732-c52f-4de8-9bdd-d43d8f919ccb-kube-api-access-hksrc\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:30.960899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.960893 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tls-certs\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:30.961036 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:30.960935 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-home\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.061591 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.061562 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-model-cache\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.061766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.061603 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-dshm\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.061766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.061627 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tmp-dir\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.061766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.061648 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksrc\" (UniqueName: \"kubernetes.io/projected/06d04732-c52f-4de8-9bdd-d43d8f919ccb-kube-api-access-hksrc\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.061766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.061673 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tls-certs\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.061766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.061707 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-home\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.062079 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.061978 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-model-cache\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.062079 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.062030 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tmp-dir\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.062261 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.062174 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-home\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.063927 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.063906 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-dshm\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.064161 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.064146 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tls-certs\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.071732 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.071690 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksrc\" (UniqueName: \"kubernetes.io/projected/06d04732-c52f-4de8-9bdd-d43d8f919ccb-kube-api-access-hksrc\") pod \"e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.147251 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.147225 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:31.278518 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:31.278493 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8"] Aug 04 00:30:31.281076 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:30:31.281048 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06d04732_c52f_4de8_9bdd_d43d8f919ccb.slice/crio-d31c06312716d9b9428a91bc48459a85812712cc14e043761a87caf36c797a31 WatchSource:0}: Error finding container d31c06312716d9b9428a91bc48459a85812712cc14e043761a87caf36c797a31: Status 404 returned error can't find the container with id d31c06312716d9b9428a91bc48459a85812712cc14e043761a87caf36c797a31 Aug 04 00:30:32.048488 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:32.048453 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" event={"ID":"06d04732-c52f-4de8-9bdd-d43d8f919ccb","Type":"ContainerStarted","Data":"c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3"} Aug 04 00:30:32.048488 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:32.048495 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" event={"ID":"06d04732-c52f-4de8-9bdd-d43d8f919ccb","Type":"ContainerStarted","Data":"d31c06312716d9b9428a91bc48459a85812712cc14e043761a87caf36c797a31"} Aug 04 00:30:32.066073 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:32.066024 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" podStartSLOduration=2.066008537 podStartE2EDuration="2.066008537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:30:32.063889618 +0000 UTC m=+1670.726799097" watchObservedRunningTime="2026-08-04 00:30:32.066008537 +0000 UTC m=+1670.728918020" Aug 04 00:30:41.148324 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:41.148289 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:41.148805 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:41.148369 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:41.161115 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:41.161094 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:42.107801 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:42.107770 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:30:51.330400 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.330363 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng"] Aug 04 00:30:51.333900 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.333877 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.336088 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.336070 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3e1ba55f-kserve-self-signed-certs\"" Aug 04 00:30:51.340296 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.340271 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng"] Aug 04 00:30:51.444089 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.444052 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-dshm\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.444264 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.444115 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-tmp-dir\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.444264 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.444139 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72558466-fed3-4605-941f-d76a3ec760b2-tls-certs\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.444264 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.444190 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczz6\" (UniqueName: \"kubernetes.io/projected/72558466-fed3-4605-941f-d76a3ec760b2-kube-api-access-vczz6\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.444422 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.444272 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-model-cache\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.444422 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.444317 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-home\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.545336 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.545301 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-dshm\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.545505 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.545349 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-tmp-dir\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.545505 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.545367 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72558466-fed3-4605-941f-d76a3ec760b2-tls-certs\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.545505 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.545399 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczz6\" (UniqueName: \"kubernetes.io/projected/72558466-fed3-4605-941f-d76a3ec760b2-kube-api-access-vczz6\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.545505 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.545430 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-model-cache\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.545505 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.545451 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-home\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.545815 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.545792 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-model-cache\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.545899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.545802 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-tmp-dir\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.545951 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.545906 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-home\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.547698 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.547674 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-dshm\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.548199 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.548164 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72558466-fed3-4605-941f-d76a3ec760b2-tls-certs\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.553178 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.553150 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczz6\" (UniqueName: \"kubernetes.io/projected/72558466-fed3-4605-941f-d76a3ec760b2-kube-api-access-vczz6\") pod \"e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.643911 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.643885 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:30:51.776073 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:51.776045 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng"] Aug 04 00:30:51.777627 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:30:51.777593 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72558466_fed3_4605_941f_d76a3ec760b2.slice/crio-165e8302b32f70382cf45946f60f94fa8d8a61c6c21546a5801e659130d3966d WatchSource:0}: Error finding container 165e8302b32f70382cf45946f60f94fa8d8a61c6c21546a5801e659130d3966d: Status 404 returned error can't find the container with id 165e8302b32f70382cf45946f60f94fa8d8a61c6c21546a5801e659130d3966d Aug 04 00:30:52.143517 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:52.143480 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" event={"ID":"72558466-fed3-4605-941f-d76a3ec760b2","Type":"ContainerStarted","Data":"dfe09ae01c120947df6333823875949984babb01e4a379f7e92c124ba6f05240"} Aug 04 00:30:52.143517 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:52.143518 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" event={"ID":"72558466-fed3-4605-941f-d76a3ec760b2","Type":"ContainerStarted","Data":"165e8302b32f70382cf45946f60f94fa8d8a61c6c21546a5801e659130d3966d"} Aug 04 00:30:52.160081 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:30:52.160033 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" podStartSLOduration=1.160017499 podStartE2EDuration="1.160017499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:30:52.158153795 +0000 UTC m=+1690.821063286" watchObservedRunningTime="2026-08-04 00:30:52.160017499 +0000 UTC m=+1690.822927003" Aug 04 00:31:01.644576 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:01.644535 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:31:01.645159 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:01.644588 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:31:01.657175 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:01.657153 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:31:02.209446 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:02.209420 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:31:27.625764 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:27.625727 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8"] Aug 04 00:31:29.189869 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:29.189819 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng"] Aug 04 00:31:42.645183 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:42.645145 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" podUID="06d04732-c52f-4de8-9bdd-d43d8f919ccb" containerName="main" containerID="cri-o://c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3" gracePeriod=45 Aug 04 00:31:42.906209 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:42.906156 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:31:43.004725 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.004689 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-model-cache\") pod \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " Aug 04 00:31:43.004921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.004778 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tmp-dir\") pod \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " Aug 04 00:31:43.004921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.004799 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tls-certs\") pod \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " Aug 04 00:31:43.004921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.004829 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-dshm\") pod \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " Aug 04 00:31:43.004921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.004880 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hksrc\" (UniqueName: \"kubernetes.io/projected/06d04732-c52f-4de8-9bdd-d43d8f919ccb-kube-api-access-hksrc\") pod \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " Aug 04 00:31:43.004921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.004910 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-home\") pod \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\" (UID: \"06d04732-c52f-4de8-9bdd-d43d8f919ccb\") " Aug 04 00:31:43.005205 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.005050 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-model-cache" (OuterVolumeSpecName: "model-cache") pod "06d04732-c52f-4de8-9bdd-d43d8f919ccb" (UID: "06d04732-c52f-4de8-9bdd-d43d8f919ccb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:31:43.005205 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.005085 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06d04732-c52f-4de8-9bdd-d43d8f919ccb" (UID: "06d04732-c52f-4de8-9bdd-d43d8f919ccb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:31:43.005300 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.005219 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-home" (OuterVolumeSpecName: "home") pod "06d04732-c52f-4de8-9bdd-d43d8f919ccb" (UID: "06d04732-c52f-4de8-9bdd-d43d8f919ccb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:31:43.005300 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.005278 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:43.005364 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.005311 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:43.007038 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.007016 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-dshm" (OuterVolumeSpecName: "dshm") pod "06d04732-c52f-4de8-9bdd-d43d8f919ccb" (UID: "06d04732-c52f-4de8-9bdd-d43d8f919ccb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:31:43.007038 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.007028 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06d04732-c52f-4de8-9bdd-d43d8f919ccb" (UID: "06d04732-c52f-4de8-9bdd-d43d8f919ccb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:31:43.007159 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.007136 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d04732-c52f-4de8-9bdd-d43d8f919ccb-kube-api-access-hksrc" (OuterVolumeSpecName: "kube-api-access-hksrc") pod "06d04732-c52f-4de8-9bdd-d43d8f919ccb" (UID: "06d04732-c52f-4de8-9bdd-d43d8f919ccb"). InnerVolumeSpecName "kube-api-access-hksrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:31:43.106649 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.106586 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d04732-c52f-4de8-9bdd-d43d8f919ccb-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:43.106649 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.106626 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:43.106649 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.106641 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hksrc\" (UniqueName: \"kubernetes.io/projected/06d04732-c52f-4de8-9bdd-d43d8f919ccb-kube-api-access-hksrc\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:43.106649 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.106656 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d04732-c52f-4de8-9bdd-d43d8f919ccb-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:43.405012 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.404976 2552 generic.go:356] "Generic (PLEG): container finished" podID="06d04732-c52f-4de8-9bdd-d43d8f919ccb" containerID="c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3" exitCode=0 Aug 04 00:31:43.405012 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.405014 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" event={"ID":"06d04732-c52f-4de8-9bdd-d43d8f919ccb","Type":"ContainerDied","Data":"c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3"} Aug 04 00:31:43.405264 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.405028 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" Aug 04 00:31:43.405264 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.405041 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8" event={"ID":"06d04732-c52f-4de8-9bdd-d43d8f919ccb","Type":"ContainerDied","Data":"d31c06312716d9b9428a91bc48459a85812712cc14e043761a87caf36c797a31"} Aug 04 00:31:43.405264 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.405058 2552 scope.go:117] "RemoveContainer" containerID="c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3" Aug 04 00:31:43.416794 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.416776 2552 scope.go:117] "RemoveContainer" containerID="c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3" Aug 04 00:31:43.417102 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:31:43.417076 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3\": container with ID starting with c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3 not found: ID does not exist" containerID="c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3" Aug 04 00:31:43.417102 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.417109 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3"} err="failed to get container status \"c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3\": rpc error: code = NotFound desc = could not find container \"c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3\": container with ID starting with c2c7de2f4a90afbefcc9131e3df076f5a9a3aa2276a42938159da963f13b4aa3 not found: ID does not exist" Aug 04 00:31:43.426812 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.426785 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8"] Aug 04 00:31:43.431213 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.431192 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3e1ba55f-kserve-766dd5c54c-qhzj8"] Aug 04 00:31:43.953571 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:43.953528 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d04732-c52f-4de8-9bdd-d43d8f919ccb" path="/var/lib/kubelet/pods/06d04732-c52f-4de8-9bdd-d43d8f919ccb/volumes" Aug 04 00:31:44.208336 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.208242 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" podUID="72558466-fed3-4605-941f-d76a3ec760b2" containerName="main" containerID="cri-o://dfe09ae01c120947df6333823875949984babb01e4a379f7e92c124ba6f05240" gracePeriod=45 Aug 04 00:31:44.415511 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.415407 2552 generic.go:356] "Generic (PLEG): container finished" podID="72558466-fed3-4605-941f-d76a3ec760b2" containerID="dfe09ae01c120947df6333823875949984babb01e4a379f7e92c124ba6f05240" exitCode=0 Aug 04 00:31:44.415683 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.415534 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" event={"ID":"72558466-fed3-4605-941f-d76a3ec760b2","Type":"ContainerDied","Data":"dfe09ae01c120947df6333823875949984babb01e4a379f7e92c124ba6f05240"} Aug 04 00:31:44.464651 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.464593 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:31:44.619586 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.619551 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-model-cache\") pod \"72558466-fed3-4605-941f-d76a3ec760b2\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " Aug 04 00:31:44.619759 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.619618 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-home\") pod \"72558466-fed3-4605-941f-d76a3ec760b2\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " Aug 04 00:31:44.619759 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.619666 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-tmp-dir\") pod \"72558466-fed3-4605-941f-d76a3ec760b2\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " Aug 04 00:31:44.619759 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.619714 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72558466-fed3-4605-941f-d76a3ec760b2-tls-certs\") pod \"72558466-fed3-4605-941f-d76a3ec760b2\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " Aug 04 00:31:44.620044 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.619763 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-dshm\") pod \"72558466-fed3-4605-941f-d76a3ec760b2\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " Aug 04 00:31:44.620044 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.619794 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vczz6\" (UniqueName: \"kubernetes.io/projected/72558466-fed3-4605-941f-d76a3ec760b2-kube-api-access-vczz6\") pod \"72558466-fed3-4605-941f-d76a3ec760b2\" (UID: \"72558466-fed3-4605-941f-d76a3ec760b2\") " Aug 04 00:31:44.620044 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.619801 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "72558466-fed3-4605-941f-d76a3ec760b2" (UID: "72558466-fed3-4605-941f-d76a3ec760b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:31:44.620044 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.620032 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72558466-fed3-4605-941f-d76a3ec760b2" (UID: "72558466-fed3-4605-941f-d76a3ec760b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:31:44.620245 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.620127 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:44.620245 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.620145 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:44.620325 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.620237 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-home" (OuterVolumeSpecName: "home") pod "72558466-fed3-4605-941f-d76a3ec760b2" (UID: "72558466-fed3-4605-941f-d76a3ec760b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:31:44.622078 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.622047 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72558466-fed3-4605-941f-d76a3ec760b2-kube-api-access-vczz6" (OuterVolumeSpecName: "kube-api-access-vczz6") pod "72558466-fed3-4605-941f-d76a3ec760b2" (UID: "72558466-fed3-4605-941f-d76a3ec760b2"). InnerVolumeSpecName "kube-api-access-vczz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:31:44.622277 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.622253 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-dshm" (OuterVolumeSpecName: "dshm") pod "72558466-fed3-4605-941f-d76a3ec760b2" (UID: "72558466-fed3-4605-941f-d76a3ec760b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:31:44.622615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.622597 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72558466-fed3-4605-941f-d76a3ec760b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72558466-fed3-4605-941f-d76a3ec760b2" (UID: "72558466-fed3-4605-941f-d76a3ec760b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:31:44.720630 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.720549 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:44.720630 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.720574 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72558466-fed3-4605-941f-d76a3ec760b2-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:44.720630 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.720583 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72558466-fed3-4605-941f-d76a3ec760b2-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:44.720630 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:44.720594 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vczz6\" (UniqueName: \"kubernetes.io/projected/72558466-fed3-4605-941f-d76a3ec760b2-kube-api-access-vczz6\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:31:45.423417 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:45.423381 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" Aug 04 00:31:45.423887 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:45.423413 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng" event={"ID":"72558466-fed3-4605-941f-d76a3ec760b2","Type":"ContainerDied","Data":"165e8302b32f70382cf45946f60f94fa8d8a61c6c21546a5801e659130d3966d"} Aug 04 00:31:45.423887 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:45.423474 2552 scope.go:117] "RemoveContainer" containerID="dfe09ae01c120947df6333823875949984babb01e4a379f7e92c124ba6f05240" Aug 04 00:31:45.443042 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:45.443016 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng"] Aug 04 00:31:45.451302 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:45.448643 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3e1ba55f-kserve-9b4dcdfc5-gglng"] Aug 04 00:31:45.953380 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:31:45.953324 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72558466-fed3-4605-941f-d76a3ec760b2" path="/var/lib/kubelet/pods/72558466-fed3-4605-941f-d76a3ec760b2/volumes" Aug 04 00:35:18.603048 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.603005 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh"] Aug 04 00:35:18.603569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.603370 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72558466-fed3-4605-941f-d76a3ec760b2" containerName="main" Aug 04 00:35:18.603569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.603381 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="72558466-fed3-4605-941f-d76a3ec760b2" containerName="main" Aug 04 00:35:18.603569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.603403 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d04732-c52f-4de8-9bdd-d43d8f919ccb" containerName="main" Aug 04 00:35:18.603569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.603410 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d04732-c52f-4de8-9bdd-d43d8f919ccb" containerName="main" Aug 04 00:35:18.603569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.603469 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d04732-c52f-4de8-9bdd-d43d8f919ccb" containerName="main" Aug 04 00:35:18.603569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.603517 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="72558466-fed3-4605-941f-d76a3ec760b2" containerName="main" Aug 04 00:35:18.606510 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.606490 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.608928 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.608905 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-418d3ab6-openshift-default-dockercfg-jl5lz\"" Aug 04 00:35:18.627558 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.627531 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh"] Aug 04 00:35:18.674204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.674163 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-token\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.674376 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.674217 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-envoy\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.674376 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.674262 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-credential-socket\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.674376 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.674293 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-socket\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.674376 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.674334 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-podinfo\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.674581 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.674451 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-data\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.674581 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.674493 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqc72\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-kube-api-access-qqc72\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.674581 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.674558 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-certs\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.674710 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.674594 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0863b1-12f5-4edc-b328-d178a2df4721-istiod-ca-cert\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775070 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775039 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-data\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775070 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775073 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqc72\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-kube-api-access-qqc72\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775311 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775109 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-certs\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775311 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775142 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0863b1-12f5-4edc-b328-d178a2df4721-istiod-ca-cert\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775311 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775181 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-token\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775311 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775211 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-envoy\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775311 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775249 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-credential-socket\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775311 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775275 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-socket\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775311 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775297 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-podinfo\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775659 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775479 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-data\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775713 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775686 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-credential-socket\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.775936 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775911 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-certs\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.776127 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775948 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0863b1-12f5-4edc-b328-d178a2df4721-istiod-ca-cert\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.776127 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.775964 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-socket\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.777623 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.777602 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-envoy\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.777880 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.777839 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-podinfo\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.784922 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.784895 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-token\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.785009 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.784973 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqc72\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-kube-api-access-qqc72\") pod \"e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:18.916907 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:18.916846 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:19.052482 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:19.052457 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh"] Aug 04 00:35:19.054345 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:35:19.054317 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0863b1_12f5_4edc_b328_d178a2df4721.slice/crio-d2952c113b5b417cdad443b0ed38fdbd0ace6217cc92e4de6ba2fe740a6eb30f WatchSource:0}: Error finding container d2952c113b5b417cdad443b0ed38fdbd0ace6217cc92e4de6ba2fe740a6eb30f: Status 404 returned error can't find the container with id d2952c113b5b417cdad443b0ed38fdbd0ace6217cc92e4de6ba2fe740a6eb30f Aug 04 00:35:19.056583 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:19.056547 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:35:19.056659 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:19.056632 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:35:19.056701 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:19.056673 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:35:19.447011 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:19.446976 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" event={"ID":"9b0863b1-12f5-4edc-b328-d178a2df4721","Type":"ContainerStarted","Data":"e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1"} Aug 04 00:35:19.447178 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:19.447029 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" event={"ID":"9b0863b1-12f5-4edc-b328-d178a2df4721","Type":"ContainerStarted","Data":"d2952c113b5b417cdad443b0ed38fdbd0ace6217cc92e4de6ba2fe740a6eb30f"} Aug 04 00:35:19.468004 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:19.467957 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" podStartSLOduration=1.467940689 podStartE2EDuration="1.467940689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:35:19.467159466 +0000 UTC m=+1958.130068945" watchObservedRunningTime="2026-08-04 00:35:19.467940689 +0000 UTC m=+1958.130850167" Aug 04 00:35:19.917811 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:19.917776 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:20.922752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:20.922722 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:21.456406 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:21.456378 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:21.457250 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:21.457227 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:22.532412 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.532383 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5"] Aug 04 00:35:22.535844 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.535825 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:22.538520 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.538497 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-418d3ab6-dockercfg-xhxm4\"" Aug 04 00:35:22.538614 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.538500 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-418d3ab6\"" Aug 04 00:35:22.546000 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.545981 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5"] Aug 04 00:35:22.611094 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.611055 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rd5k\" (UniqueName: \"kubernetes.io/projected/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-kube-api-access-7rd5k\") pod \"payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5\" (UID: \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:22.611277 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.611109 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5\" (UID: \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:22.712437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.712398 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rd5k\" (UniqueName: \"kubernetes.io/projected/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-kube-api-access-7rd5k\") pod \"payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5\" (UID: \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:22.712437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.712441 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5\" (UID: \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:22.713212 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.713193 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5\" (UID: \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:22.720506 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.720480 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rd5k\" (UniqueName: \"kubernetes.io/projected/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-kube-api-access-7rd5k\") pod \"payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5\" (UID: \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:22.847039 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.846943 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:22.984584 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:22.984553 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5"] Aug 04 00:35:22.987959 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:35:22.987927 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbe3a02_8c4c_40d5_b76a_5103709b1bdc.slice/crio-b5f834f0428ec9b9f446c5441e0107ff34f5534695ca2fb9f659045d77863eec WatchSource:0}: Error finding container b5f834f0428ec9b9f446c5441e0107ff34f5534695ca2fb9f659045d77863eec: Status 404 returned error can't find the container with id b5f834f0428ec9b9f446c5441e0107ff34f5534695ca2fb9f659045d77863eec Aug 04 00:35:23.466885 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:23.466839 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" event={"ID":"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc","Type":"ContainerStarted","Data":"4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036"} Aug 04 00:35:23.467068 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:23.466901 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" event={"ID":"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc","Type":"ContainerStarted","Data":"b5f834f0428ec9b9f446c5441e0107ff34f5534695ca2fb9f659045d77863eec"} Aug 04 00:35:23.467133 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:23.467104 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:23.482950 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:23.482895 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" podStartSLOduration=1.482878159 podStartE2EDuration="1.482878159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:35:23.481582021 +0000 UTC m=+1962.144491499" watchObservedRunningTime="2026-08-04 00:35:23.482878159 +0000 UTC m=+1962.145787631" Aug 04 00:35:34.474897 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:34.474846 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:34.529997 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:34.529968 2552 generic.go:356] "Generic (PLEG): container finished" podID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerID="e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1" exitCode=0 Aug 04 00:35:34.530109 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:34.530005 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" event={"ID":"9b0863b1-12f5-4edc-b328-d178a2df4721","Type":"ContainerDied","Data":"e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1"} Aug 04 00:35:34.530419 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:34.530403 2552 scope.go:117] "RemoveContainer" containerID="e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1" Aug 04 00:35:34.531508 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:34.531478 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:35:34.531614 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:34.531561 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:35:34.531672 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:34.531611 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:35:35.536754 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:35.536630 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" event={"ID":"9b0863b1-12f5-4edc-b328-d178a2df4721","Type":"ContainerStarted","Data":"e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8"} Aug 04 00:35:35.917352 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:35.917307 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:35.918561 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:35.918539 2552 patch_prober.go:28] interesting pod/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 04 00:35:35.918631 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:35.918598 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 04 00:35:36.456680 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:36.456647 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:36.917883 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:36.917821 2552 patch_prober.go:28] interesting pod/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 04 00:35:36.918298 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:36.917910 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 04 00:35:37.917335 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:37.917299 2552 patch_prober.go:28] interesting pod/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 04 00:35:37.917494 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:37.917358 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 04 00:35:38.917342 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:38.917309 2552 patch_prober.go:28] interesting pod/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 04 00:35:38.917734 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:38.917365 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 04 00:35:39.917417 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:39.917379 2552 patch_prober.go:28] interesting pod/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 04 00:35:39.917812 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:39.917446 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 04 00:35:40.921286 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:40.921257 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:40.922218 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:40.922201 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:46.396127 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.396090 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4"] Aug 04 00:35:46.399385 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.399368 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" Aug 04 00:35:46.401733 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.401711 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:35:46.401833 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.401711 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 00:35:46.407624 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.407604 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4"] Aug 04 00:35:46.536737 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.536693 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84035b1-d59a-4650-8519-c23bccd3f3bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4\" (UID: \"b84035b1-d59a-4650-8519-c23bccd3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" Aug 04 00:35:46.637844 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.637812 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84035b1-d59a-4650-8519-c23bccd3f3bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4\" (UID: \"b84035b1-d59a-4650-8519-c23bccd3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" Aug 04 00:35:46.638424 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.638404 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84035b1-d59a-4650-8519-c23bccd3f3bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4\" (UID: \"b84035b1-d59a-4650-8519-c23bccd3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" Aug 04 00:35:46.711503 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.711429 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" Aug 04 00:35:46.842574 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:46.842552 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4"] Aug 04 00:35:46.844447 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:35:46.844418 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84035b1_d59a_4650_8519_c23bccd3f3bd.slice/crio-e07285e0054ddb47bf19ef63fa4f1e76defaeaf5487c7f636a79f3f3e1163fec WatchSource:0}: Error finding container e07285e0054ddb47bf19ef63fa4f1e76defaeaf5487c7f636a79f3f3e1163fec: Status 404 returned error can't find the container with id e07285e0054ddb47bf19ef63fa4f1e76defaeaf5487c7f636a79f3f3e1163fec Aug 04 00:35:47.600378 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:47.600347 2552 generic.go:356] "Generic (PLEG): container finished" podID="b84035b1-d59a-4650-8519-c23bccd3f3bd" containerID="bb93660cc8003ee1ae04c54a26b1ecdade30f9fa443cdf38e791b9ebad555bef" exitCode=0 Aug 04 00:35:47.600759 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:47.600438 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" event={"ID":"b84035b1-d59a-4650-8519-c23bccd3f3bd","Type":"ContainerDied","Data":"bb93660cc8003ee1ae04c54a26b1ecdade30f9fa443cdf38e791b9ebad555bef"} Aug 04 00:35:47.600759 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:47.600503 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" event={"ID":"b84035b1-d59a-4650-8519-c23bccd3f3bd","Type":"ContainerStarted","Data":"e07285e0054ddb47bf19ef63fa4f1e76defaeaf5487c7f636a79f3f3e1163fec"} Aug 04 00:35:48.755413 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:48.755389 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" Aug 04 00:35:48.859400 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:48.859330 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84035b1-d59a-4650-8519-c23bccd3f3bd-maas-api-service-ca\") pod \"b84035b1-d59a-4650-8519-c23bccd3f3bd\" (UID: \"b84035b1-d59a-4650-8519-c23bccd3f3bd\") " Aug 04 00:35:48.859653 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:48.859631 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84035b1-d59a-4650-8519-c23bccd3f3bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b84035b1-d59a-4650-8519-c23bccd3f3bd" (UID: "b84035b1-d59a-4650-8519-c23bccd3f3bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:35:48.960506 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:48.960479 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84035b1-d59a-4650-8519-c23bccd3f3bd-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:49.612194 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:49.612165 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" Aug 04 00:35:49.612365 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:49.612165 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4" event={"ID":"b84035b1-d59a-4650-8519-c23bccd3f3bd","Type":"ContainerDied","Data":"e07285e0054ddb47bf19ef63fa4f1e76defaeaf5487c7f636a79f3f3e1163fec"} Aug 04 00:35:49.612365 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:49.612289 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e07285e0054ddb47bf19ef63fa4f1e76defaeaf5487c7f636a79f3f3e1163fec" Aug 04 00:35:52.666736 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:52.666699 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5"] Aug 04 00:35:52.667213 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:52.666931 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" podUID="2dbe3a02-8c4c-40d5-b76a-5103709b1bdc" containerName="payload-pre-processing" containerID="cri-o://4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036" gracePeriod=30 Aug 04 00:35:52.923452 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:52.923358 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4"] Aug 04 00:35:52.926528 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:52.926501 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-418d3ab6-q6kh4"] Aug 04 00:35:52.933131 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:52.933106 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:52.998166 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:52.998129 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-plugins-config-volume\") pod \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\" (UID: \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\") " Aug 04 00:35:52.998362 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:52.998197 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rd5k\" (UniqueName: \"kubernetes.io/projected/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-kube-api-access-7rd5k\") pod \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\" (UID: \"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc\") " Aug 04 00:35:52.998538 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:52.998500 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2dbe3a02-8c4c-40d5-b76a-5103709b1bdc" (UID: "2dbe3a02-8c4c-40d5-b76a-5103709b1bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:35:53.000356 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.000325 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-kube-api-access-7rd5k" (OuterVolumeSpecName: "kube-api-access-7rd5k") pod "2dbe3a02-8c4c-40d5-b76a-5103709b1bdc" (UID: "2dbe3a02-8c4c-40d5-b76a-5103709b1bdc"). InnerVolumeSpecName "kube-api-access-7rd5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:35:53.045013 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.044972 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh"] Aug 04 00:35:53.045251 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.045223 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" containerID="cri-o://e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8" gracePeriod=30 Aug 04 00:35:53.099693 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.099662 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rd5k\" (UniqueName: \"kubernetes.io/projected/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-kube-api-access-7rd5k\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:53.099693 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.099685 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:53.631921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.631884 2552 generic.go:356] "Generic (PLEG): container finished" podID="2dbe3a02-8c4c-40d5-b76a-5103709b1bdc" containerID="4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036" exitCode=0 Aug 04 00:35:53.632078 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.631928 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" Aug 04 00:35:53.632078 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.631967 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" event={"ID":"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc","Type":"ContainerDied","Data":"4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036"} Aug 04 00:35:53.632078 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.632020 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5" event={"ID":"2dbe3a02-8c4c-40d5-b76a-5103709b1bdc","Type":"ContainerDied","Data":"b5f834f0428ec9b9f446c5441e0107ff34f5534695ca2fb9f659045d77863eec"} Aug 04 00:35:53.632078 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.632041 2552 scope.go:117] "RemoveContainer" containerID="4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036" Aug 04 00:35:53.640797 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.640780 2552 scope.go:117] "RemoveContainer" containerID="4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036" Aug 04 00:35:53.641080 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:35:53.641060 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036\": container with ID starting with 4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036 not found: ID does not exist" containerID="4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036" Aug 04 00:35:53.641153 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.641087 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036"} err="failed to get container status \"4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036\": rpc error: code = NotFound desc = could not find container \"4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036\": container with ID starting with 4ce96f94090bebf8ee265b9acdb2c03020bf6dbaca72cdbd05a52f5e1bb18036 not found: ID does not exist" Aug 04 00:35:53.653182 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.653146 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5"] Aug 04 00:35:53.655030 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.655011 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-418d3ab6-569bbb58b8-x5gg5"] Aug 04 00:35:53.953267 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.953189 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbe3a02-8c4c-40d5-b76a-5103709b1bdc" path="/var/lib/kubelet/pods/2dbe3a02-8c4c-40d5-b76a-5103709b1bdc/volumes" Aug 04 00:35:53.953608 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:53.953544 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b84035b1-d59a-4650-8519-c23bccd3f3bd" path="/var/lib/kubelet/pods/b84035b1-d59a-4650-8519-c23bccd3f3bd/volumes" Aug 04 00:35:54.056710 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.056675 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z"] Aug 04 00:35:54.057051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.057033 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b84035b1-d59a-4650-8519-c23bccd3f3bd" containerName="revoke-keys" Aug 04 00:35:54.057051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.057048 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b84035b1-d59a-4650-8519-c23bccd3f3bd" containerName="revoke-keys" Aug 04 00:35:54.057203 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.057071 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbe3a02-8c4c-40d5-b76a-5103709b1bdc" containerName="payload-pre-processing" Aug 04 00:35:54.057203 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.057078 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbe3a02-8c4c-40d5-b76a-5103709b1bdc" containerName="payload-pre-processing" Aug 04 00:35:54.057203 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.057142 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b84035b1-d59a-4650-8519-c23bccd3f3bd" containerName="revoke-keys" Aug 04 00:35:54.057203 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.057153 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbe3a02-8c4c-40d5-b76a-5103709b1bdc" containerName="payload-pre-processing" Aug 04 00:35:54.061465 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.061447 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.064077 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.064058 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ae02d212-gw-openshift-default-dockercfg-l99j6\"" Aug 04 00:35:54.070584 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.070559 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z"] Aug 04 00:35:54.108292 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.108268 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-certs\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.108434 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.108322 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-socket\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.108434 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.108339 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-envoy\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.108434 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.108373 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-podinfo\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.108535 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.108443 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78037fc-b597-4b3c-a893-c9b404767ff6-istiod-ca-cert\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.108535 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.108477 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-token\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.108535 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.108494 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-data\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.108535 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.108511 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltfct\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-kube-api-access-ltfct\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.108535 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.108533 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-credential-socket\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210009 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.209929 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-socket\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210009 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.209963 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-envoy\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210009 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.209983 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-podinfo\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210291 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210109 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78037fc-b597-4b3c-a893-c9b404767ff6-istiod-ca-cert\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210291 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210182 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-token\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210291 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210214 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-data\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210291 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210245 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltfct\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-kube-api-access-ltfct\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210499 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210436 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-credential-socket\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210499 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210449 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-socket\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210607 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210520 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-certs\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210607 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210547 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-data\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210804 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210782 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-credential-socket\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.210985 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210964 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78037fc-b597-4b3c-a893-c9b404767ff6-istiod-ca-cert\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.211054 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.210998 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-certs\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.212518 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.212499 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-envoy\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.212616 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.212599 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-podinfo\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.218663 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.218641 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-token\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.218800 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.218782 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltfct\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-kube-api-access-ltfct\") pod \"e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.373092 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.373055 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:54.512998 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.512969 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z"] Aug 04 00:35:54.514927 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:35:54.514899 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78037fc_b597_4b3c_a893_c9b404767ff6.slice/crio-a5eaa3248eda1a4aa09b68d25a159df3b10ca27c5b7dd04f67c2d463c9eb59c1 WatchSource:0}: Error finding container a5eaa3248eda1a4aa09b68d25a159df3b10ca27c5b7dd04f67c2d463c9eb59c1: Status 404 returned error can't find the container with id a5eaa3248eda1a4aa09b68d25a159df3b10ca27c5b7dd04f67c2d463c9eb59c1 Aug 04 00:35:54.517104 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.517064 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:35:54.517172 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.517155 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:35:54.517214 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.517199 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:35:54.639274 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.639237 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" event={"ID":"d78037fc-b597-4b3c-a893-c9b404767ff6","Type":"ContainerStarted","Data":"0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd"} Aug 04 00:35:54.639462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.639303 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" event={"ID":"d78037fc-b597-4b3c-a893-c9b404767ff6","Type":"ContainerStarted","Data":"a5eaa3248eda1a4aa09b68d25a159df3b10ca27c5b7dd04f67c2d463c9eb59c1"} Aug 04 00:35:54.658371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:54.658323 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" podStartSLOduration=0.658304923 podStartE2EDuration="658.304923ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:35:54.656469906 +0000 UTC m=+1993.319379386" watchObservedRunningTime="2026-08-04 00:35:54.658304923 +0000 UTC m=+1993.321214401" Aug 04 00:35:55.373950 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:55.373919 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:55.379387 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:55.379360 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:55.646474 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:55.646395 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:55.647374 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:55.647354 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:35:55.924266 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:55.924184 2552 patch_prober.go:28] interesting pod/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 00:35:55.924266 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:55.924240 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:35:58.298109 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.298083 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:58.350127 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350055 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-token\") pod \"9b0863b1-12f5-4edc-b328-d178a2df4721\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " Aug 04 00:35:58.350127 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350088 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-envoy\") pod \"9b0863b1-12f5-4edc-b328-d178a2df4721\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " Aug 04 00:35:58.350127 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350108 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-certs\") pod \"9b0863b1-12f5-4edc-b328-d178a2df4721\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " Aug 04 00:35:58.350436 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350184 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-socket\") pod \"9b0863b1-12f5-4edc-b328-d178a2df4721\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " Aug 04 00:35:58.350436 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350203 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqc72\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-kube-api-access-qqc72\") pod \"9b0863b1-12f5-4edc-b328-d178a2df4721\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " Aug 04 00:35:58.350436 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350248 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-credential-socket\") pod \"9b0863b1-12f5-4edc-b328-d178a2df4721\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " Aug 04 00:35:58.350436 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350288 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-podinfo\") pod \"9b0863b1-12f5-4edc-b328-d178a2df4721\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " Aug 04 00:35:58.350436 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350326 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-data\") pod \"9b0863b1-12f5-4edc-b328-d178a2df4721\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " Aug 04 00:35:58.350436 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350361 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0863b1-12f5-4edc-b328-d178a2df4721-istiod-ca-cert\") pod \"9b0863b1-12f5-4edc-b328-d178a2df4721\" (UID: \"9b0863b1-12f5-4edc-b328-d178a2df4721\") " Aug 04 00:35:58.350737 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350458 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b0863b1-12f5-4edc-b328-d178a2df4721" (UID: "9b0863b1-12f5-4edc-b328-d178a2df4721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:35:58.350737 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350483 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b0863b1-12f5-4edc-b328-d178a2df4721" (UID: "9b0863b1-12f5-4edc-b328-d178a2df4721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:35:58.350737 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350550 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b0863b1-12f5-4edc-b328-d178a2df4721" (UID: "9b0863b1-12f5-4edc-b328-d178a2df4721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:35:58.350916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350764 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:58.350916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350783 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:58.350916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350797 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:58.350916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350837 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0863b1-12f5-4edc-b328-d178a2df4721-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b0863b1-12f5-4edc-b328-d178a2df4721" (UID: "9b0863b1-12f5-4edc-b328-d178a2df4721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:35:58.350916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.350835 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b0863b1-12f5-4edc-b328-d178a2df4721" (UID: "9b0863b1-12f5-4edc-b328-d178a2df4721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:35:58.352442 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.352417 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b0863b1-12f5-4edc-b328-d178a2df4721" (UID: "9b0863b1-12f5-4edc-b328-d178a2df4721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:35:58.352638 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.352621 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b0863b1-12f5-4edc-b328-d178a2df4721" (UID: "9b0863b1-12f5-4edc-b328-d178a2df4721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:35:58.353009 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.352987 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-kube-api-access-qqc72" (OuterVolumeSpecName: "kube-api-access-qqc72") pod "9b0863b1-12f5-4edc-b328-d178a2df4721" (UID: "9b0863b1-12f5-4edc-b328-d178a2df4721"). InnerVolumeSpecName "kube-api-access-qqc72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:35:58.353009 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.352994 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b0863b1-12f5-4edc-b328-d178a2df4721" (UID: "9b0863b1-12f5-4edc-b328-d178a2df4721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:35:58.451559 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.451529 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:58.451559 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.451554 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:58.451559 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.451563 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0863b1-12f5-4edc-b328-d178a2df4721-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:58.451774 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.451571 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:58.451774 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.451579 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0863b1-12f5-4edc-b328-d178a2df4721-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:58.451774 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.451587 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqc72\" (UniqueName: \"kubernetes.io/projected/9b0863b1-12f5-4edc-b328-d178a2df4721-kube-api-access-qqc72\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:35:58.662594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.662560 2552 generic.go:356] "Generic (PLEG): container finished" podID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerID="e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8" exitCode=0 Aug 04 00:35:58.662769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.662610 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" Aug 04 00:35:58.662769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.662616 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" event={"ID":"9b0863b1-12f5-4edc-b328-d178a2df4721","Type":"ContainerDied","Data":"e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8"} Aug 04 00:35:58.662769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.662658 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh" event={"ID":"9b0863b1-12f5-4edc-b328-d178a2df4721","Type":"ContainerDied","Data":"d2952c113b5b417cdad443b0ed38fdbd0ace6217cc92e4de6ba2fe740a6eb30f"} Aug 04 00:35:58.662769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.662680 2552 scope.go:117] "RemoveContainer" containerID="e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8" Aug 04 00:35:58.672262 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.672240 2552 scope.go:117] "RemoveContainer" containerID="e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1" Aug 04 00:35:58.679950 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.679934 2552 scope.go:117] "RemoveContainer" containerID="e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8" Aug 04 00:35:58.680225 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:35:58.680204 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8\": container with ID starting with e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8 not found: ID does not exist" containerID="e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8" Aug 04 00:35:58.680281 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.680238 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8"} err="failed to get container status \"e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8\": rpc error: code = NotFound desc = could not find container \"e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8\": container with ID starting with e8d4f3986dbd8bc096c61cf63e72d10fe6fee74fdf09ccd27c6029b4b7d7b6a8 not found: ID does not exist" Aug 04 00:35:58.680281 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.680259 2552 scope.go:117] "RemoveContainer" containerID="e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1" Aug 04 00:35:58.680481 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:35:58.680464 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1\": container with ID starting with e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1 not found: ID does not exist" containerID="e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1" Aug 04 00:35:58.680525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.680484 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1"} err="failed to get container status \"e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1\": rpc error: code = NotFound desc = could not find container \"e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1\": container with ID starting with e7980089539bf655393b6543118a8d7de629608de23e17fc6cbafc89b34180e1 not found: ID does not exist" Aug 04 00:35:58.686915 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.686889 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh"] Aug 04 00:35:58.692768 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:58.692748 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-418d3ab6-openshift-default-867fcf86f4-xp7bh"] Aug 04 00:35:59.958513 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:35:59.958481 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" path="/var/lib/kubelet/pods/9b0863b1-12f5-4edc-b328-d178a2df4721/volumes" Aug 04 00:36:08.722100 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:08.722017 2552 generic.go:356] "Generic (PLEG): container finished" podID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerID="0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd" exitCode=0 Aug 04 00:36:08.722509 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:08.722089 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" event={"ID":"d78037fc-b597-4b3c-a893-c9b404767ff6","Type":"ContainerDied","Data":"0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd"} Aug 04 00:36:08.722509 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:08.722482 2552 scope.go:117] "RemoveContainer" containerID="0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd" Aug 04 00:36:08.723449 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:08.723426 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:36:08.723500 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:08.723489 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:36:08.723540 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:08.723520 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:36:09.728383 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:09.728347 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" event={"ID":"d78037fc-b597-4b3c-a893-c9b404767ff6","Type":"ContainerStarted","Data":"6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62"} Aug 04 00:36:10.374808 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:10.374389 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:36:10.375680 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:10.375642 2552 patch_prober.go:28] interesting pod/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 00:36:10.375801 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:10.375706 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 00:36:10.647012 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:10.646929 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:36:11.374183 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:11.373691 2552 patch_prober.go:28] interesting pod/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 00:36:11.374183 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:11.373755 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 00:36:12.373833 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:12.373799 2552 patch_prober.go:28] interesting pod/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 00:36:12.374112 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:12.374081 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 00:36:13.374502 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:13.374459 2552 patch_prober.go:28] interesting pod/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 00:36:13.375159 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:13.375118 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 00:36:14.373825 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:14.373781 2552 patch_prober.go:28] interesting pod/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 00:36:14.374132 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:14.374096 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 00:36:15.378437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:15.378408 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:36:15.379479 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:15.379456 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:36:21.094479 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.094439 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v"] Aug 04 00:36:21.094967 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.094812 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" Aug 04 00:36:21.094967 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.094824 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" Aug 04 00:36:21.094967 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.094832 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" Aug 04 00:36:21.094967 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.094837 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" Aug 04 00:36:21.094967 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.094926 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" Aug 04 00:36:21.094967 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.094936 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0863b1-12f5-4edc-b328-d178a2df4721" containerName="istio-proxy" Aug 04 00:36:21.098031 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.098016 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" Aug 04 00:36:21.100460 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.100435 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:36:21.100588 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.100511 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 00:36:21.105302 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.105278 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v"] Aug 04 00:36:21.167625 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.167585 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fe2ce2-d0b7-43c6-90dc-e78784bd6696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v\" (UID: \"f4fe2ce2-d0b7-43c6-90dc-e78784bd6696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" Aug 04 00:36:21.268531 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.268495 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fe2ce2-d0b7-43c6-90dc-e78784bd6696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v\" (UID: \"f4fe2ce2-d0b7-43c6-90dc-e78784bd6696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" Aug 04 00:36:21.269150 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.269128 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fe2ce2-d0b7-43c6-90dc-e78784bd6696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v\" (UID: \"f4fe2ce2-d0b7-43c6-90dc-e78784bd6696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" Aug 04 00:36:21.410331 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.410295 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" Aug 04 00:36:21.541750 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.541725 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v"] Aug 04 00:36:21.543797 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:36:21.543766 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fe2ce2_d0b7_43c6_90dc_e78784bd6696.slice/crio-ff4ebfe9ec018a80c53294f9c708f68260236709e14fbdad04de9be4a40ddf51 WatchSource:0}: Error finding container ff4ebfe9ec018a80c53294f9c708f68260236709e14fbdad04de9be4a40ddf51: Status 404 returned error can't find the container with id ff4ebfe9ec018a80c53294f9c708f68260236709e14fbdad04de9be4a40ddf51 Aug 04 00:36:21.794374 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.794343 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" event={"ID":"f4fe2ce2-d0b7-43c6-90dc-e78784bd6696","Type":"ContainerStarted","Data":"86d4849323761bb39100fb162ba9cf18f96f8f44f56d882e06226db295be09c3"} Aug 04 00:36:21.794495 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.794395 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" event={"ID":"f4fe2ce2-d0b7-43c6-90dc-e78784bd6696","Type":"ContainerStarted","Data":"ff4ebfe9ec018a80c53294f9c708f68260236709e14fbdad04de9be4a40ddf51"} Aug 04 00:36:21.810849 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:21.810800 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" podStartSLOduration=0.810783855 podStartE2EDuration="810.783855ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:36:21.80891191 +0000 UTC m=+2020.471821389" watchObservedRunningTime="2026-08-04 00:36:21.810783855 +0000 UTC m=+2020.473693363" Aug 04 00:36:22.801073 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:22.801039 2552 generic.go:356] "Generic (PLEG): container finished" podID="f4fe2ce2-d0b7-43c6-90dc-e78784bd6696" containerID="86d4849323761bb39100fb162ba9cf18f96f8f44f56d882e06226db295be09c3" exitCode=0 Aug 04 00:36:22.801465 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:22.801136 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" event={"ID":"f4fe2ce2-d0b7-43c6-90dc-e78784bd6696","Type":"ContainerDied","Data":"86d4849323761bb39100fb162ba9cf18f96f8f44f56d882e06226db295be09c3"} Aug 04 00:36:23.939940 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:23.939915 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" Aug 04 00:36:24.090836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:24.090738 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fe2ce2-d0b7-43c6-90dc-e78784bd6696-maas-api-service-ca\") pod \"f4fe2ce2-d0b7-43c6-90dc-e78784bd6696\" (UID: \"f4fe2ce2-d0b7-43c6-90dc-e78784bd6696\") " Aug 04 00:36:24.091196 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:24.091167 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fe2ce2-d0b7-43c6-90dc-e78784bd6696-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4fe2ce2-d0b7-43c6-90dc-e78784bd6696" (UID: "f4fe2ce2-d0b7-43c6-90dc-e78784bd6696"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:36:24.191899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:24.191838 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fe2ce2-d0b7-43c6-90dc-e78784bd6696-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:24.812714 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:24.812684 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" Aug 04 00:36:24.812919 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:24.812730 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v" event={"ID":"f4fe2ce2-d0b7-43c6-90dc-e78784bd6696","Type":"ContainerDied","Data":"ff4ebfe9ec018a80c53294f9c708f68260236709e14fbdad04de9be4a40ddf51"} Aug 04 00:36:24.812919 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:24.812783 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff4ebfe9ec018a80c53294f9c708f68260236709e14fbdad04de9be4a40ddf51" Aug 04 00:36:27.799527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:27.799490 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v"] Aug 04 00:36:27.802297 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:27.802272 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ae02d212-dgk2v"] Aug 04 00:36:27.923645 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:27.923614 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z"] Aug 04 00:36:27.923885 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:27.923816 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" containerID="cri-o://6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62" gracePeriod=30 Aug 04 00:36:27.952494 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:27.952462 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4fe2ce2-d0b7-43c6-90dc-e78784bd6696" path="/var/lib/kubelet/pods/f4fe2ce2-d0b7-43c6-90dc-e78784bd6696/volumes" Aug 04 00:36:28.633100 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.633067 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858"] Aug 04 00:36:28.633509 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.633490 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4fe2ce2-d0b7-43c6-90dc-e78784bd6696" containerName="revoke-keys" Aug 04 00:36:28.633616 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.633511 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4fe2ce2-d0b7-43c6-90dc-e78784bd6696" containerName="revoke-keys" Aug 04 00:36:28.633670 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.633628 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4fe2ce2-d0b7-43c6-90dc-e78784bd6696" containerName="revoke-keys" Aug 04 00:36:28.636934 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.636912 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.639192 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.639174 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-62613146-openshift-default-dockercfg-rhj5n\"" Aug 04 00:36:28.648617 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.648590 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858"] Aug 04 00:36:28.733284 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.733234 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-socket\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.733284 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.733289 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-envoy\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.733525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.733331 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-istio-token\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.733525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.733373 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-data\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.733525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.733402 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-certs\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.733525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.733442 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq2l6\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-kube-api-access-bq2l6\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.733525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.733470 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25dad10-72e1-4baf-847d-a49815c60216-istio-podinfo\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.733525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.733502 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-credential-socket\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.733525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.733524 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25dad10-72e1-4baf-847d-a49815c60216-istiod-ca-cert\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834019 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.833936 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-data\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834019 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.833978 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-certs\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834019 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834015 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq2l6\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-kube-api-access-bq2l6\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834032 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25dad10-72e1-4baf-847d-a49815c60216-istio-podinfo\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834063 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-credential-socket\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834078 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25dad10-72e1-4baf-847d-a49815c60216-istiod-ca-cert\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834117 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-socket\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834258 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-envoy\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834335 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-istio-token\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834427 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-data\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834552 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834504 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-certs\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834923 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834657 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-socket\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.834923 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834755 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-credential-socket\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.835002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.834978 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25dad10-72e1-4baf-847d-a49815c60216-istiod-ca-cert\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.836587 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.836565 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-envoy\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.836725 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.836707 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25dad10-72e1-4baf-847d-a49815c60216-istio-podinfo\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.841303 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.841274 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-istio-token\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.841409 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.841357 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq2l6\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-kube-api-access-bq2l6\") pod \"e2e-ait-62613146-openshift-default-5cd7f87474-tt858\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:28.947574 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:28.947543 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:29.083359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:29.083299 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858"] Aug 04 00:36:29.085649 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:36:29.085587 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25dad10_72e1_4baf_847d_a49815c60216.slice/crio-0710c51a942cfe13c7d6765e6b30d93b94f66165e2686f9b87b44947b2a586a6 WatchSource:0}: Error finding container 0710c51a942cfe13c7d6765e6b30d93b94f66165e2686f9b87b44947b2a586a6: Status 404 returned error can't find the container with id 0710c51a942cfe13c7d6765e6b30d93b94f66165e2686f9b87b44947b2a586a6 Aug 04 00:36:29.087795 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:29.087762 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:36:29.088180 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:29.087828 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:36:29.088180 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:29.087884 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:36:29.839728 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:29.839692 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" event={"ID":"f25dad10-72e1-4baf-847d-a49815c60216","Type":"ContainerStarted","Data":"af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6"} Aug 04 00:36:29.840129 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:29.839746 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" event={"ID":"f25dad10-72e1-4baf-847d-a49815c60216","Type":"ContainerStarted","Data":"0710c51a942cfe13c7d6765e6b30d93b94f66165e2686f9b87b44947b2a586a6"} Aug 04 00:36:29.862838 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:29.862790 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" podStartSLOduration=1.862773783 podStartE2EDuration="1.862773783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:36:29.861276827 +0000 UTC m=+2028.524186307" watchObservedRunningTime="2026-08-04 00:36:29.862773783 +0000 UTC m=+2028.525683261" Aug 04 00:36:29.952452 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:29.952416 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:30.381903 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:30.381848 2552 patch_prober.go:28] interesting pod/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 00:36:30.382077 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:30.381931 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:36:30.953294 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:30.953262 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:31.849177 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:31.849143 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:31.850092 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:31.850072 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:33.186847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.186783 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:36:33.275975 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.275939 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-envoy\") pod \"d78037fc-b597-4b3c-a893-c9b404767ff6\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " Aug 04 00:36:33.275975 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.275978 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltfct\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-kube-api-access-ltfct\") pod \"d78037fc-b597-4b3c-a893-c9b404767ff6\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " Aug 04 00:36:33.276212 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276001 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-token\") pod \"d78037fc-b597-4b3c-a893-c9b404767ff6\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " Aug 04 00:36:33.276212 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276031 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-certs\") pod \"d78037fc-b597-4b3c-a893-c9b404767ff6\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " Aug 04 00:36:33.276212 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276087 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-data\") pod \"d78037fc-b597-4b3c-a893-c9b404767ff6\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " Aug 04 00:36:33.276212 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276169 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-socket\") pod \"d78037fc-b597-4b3c-a893-c9b404767ff6\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " Aug 04 00:36:33.276212 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276199 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-podinfo\") pod \"d78037fc-b597-4b3c-a893-c9b404767ff6\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " Aug 04 00:36:33.276457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276234 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78037fc-b597-4b3c-a893-c9b404767ff6-istiod-ca-cert\") pod \"d78037fc-b597-4b3c-a893-c9b404767ff6\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " Aug 04 00:36:33.276457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276267 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-credential-socket\") pod \"d78037fc-b597-4b3c-a893-c9b404767ff6\" (UID: \"d78037fc-b597-4b3c-a893-c9b404767ff6\") " Aug 04 00:36:33.276457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276438 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-data" (OuterVolumeSpecName: "istio-data") pod "d78037fc-b597-4b3c-a893-c9b404767ff6" (UID: "d78037fc-b597-4b3c-a893-c9b404767ff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:36:33.276591 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276465 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d78037fc-b597-4b3c-a893-c9b404767ff6" (UID: "d78037fc-b597-4b3c-a893-c9b404767ff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:36:33.276647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276603 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:33.276647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276611 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d78037fc-b597-4b3c-a893-c9b404767ff6" (UID: "d78037fc-b597-4b3c-a893-c9b404767ff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:36:33.276647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276624 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:33.276647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276633 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78037fc-b597-4b3c-a893-c9b404767ff6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d78037fc-b597-4b3c-a893-c9b404767ff6" (UID: "d78037fc-b597-4b3c-a893-c9b404767ff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:36:33.276886 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.276701 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d78037fc-b597-4b3c-a893-c9b404767ff6" (UID: "d78037fc-b597-4b3c-a893-c9b404767ff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:36:33.278206 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.278182 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d78037fc-b597-4b3c-a893-c9b404767ff6" (UID: "d78037fc-b597-4b3c-a893-c9b404767ff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:36:33.278528 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.278498 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-kube-api-access-ltfct" (OuterVolumeSpecName: "kube-api-access-ltfct") pod "d78037fc-b597-4b3c-a893-c9b404767ff6" (UID: "d78037fc-b597-4b3c-a893-c9b404767ff6"). InnerVolumeSpecName "kube-api-access-ltfct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:36:33.278647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.278530 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-token" (OuterVolumeSpecName: "istio-token") pod "d78037fc-b597-4b3c-a893-c9b404767ff6" (UID: "d78037fc-b597-4b3c-a893-c9b404767ff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:36:33.278647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.278579 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d78037fc-b597-4b3c-a893-c9b404767ff6" (UID: "d78037fc-b597-4b3c-a893-c9b404767ff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:36:33.377274 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.377243 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:33.377274 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.377270 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:33.377445 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.377281 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltfct\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-kube-api-access-ltfct\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:33.377445 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.377291 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:33.377445 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.377299 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78037fc-b597-4b3c-a893-c9b404767ff6-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:33.377445 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.377307 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78037fc-b597-4b3c-a893-c9b404767ff6-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:33.377445 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.377315 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78037fc-b597-4b3c-a893-c9b404767ff6-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:36:33.859787 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.859757 2552 generic.go:356] "Generic (PLEG): container finished" podID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerID="6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62" exitCode=0 Aug 04 00:36:33.860020 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.859804 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" Aug 04 00:36:33.860020 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.859852 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" event={"ID":"d78037fc-b597-4b3c-a893-c9b404767ff6","Type":"ContainerDied","Data":"6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62"} Aug 04 00:36:33.860020 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.859937 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z" event={"ID":"d78037fc-b597-4b3c-a893-c9b404767ff6","Type":"ContainerDied","Data":"a5eaa3248eda1a4aa09b68d25a159df3b10ca27c5b7dd04f67c2d463c9eb59c1"} Aug 04 00:36:33.860020 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.859968 2552 scope.go:117] "RemoveContainer" containerID="6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62" Aug 04 00:36:33.869307 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.869287 2552 scope.go:117] "RemoveContainer" containerID="0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd" Aug 04 00:36:33.878430 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.878413 2552 scope.go:117] "RemoveContainer" containerID="6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62" Aug 04 00:36:33.878690 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:36:33.878667 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62\": container with ID starting with 6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62 not found: ID does not exist" containerID="6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62" Aug 04 00:36:33.878836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.878700 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62"} err="failed to get container status \"6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62\": rpc error: code = NotFound desc = could not find container \"6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62\": container with ID starting with 6db1a463797aa6b4e4da1a561462f5f6282df4b3f479f3f288efe9507283db62 not found: ID does not exist" Aug 04 00:36:33.878836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.878724 2552 scope.go:117] "RemoveContainer" containerID="0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd" Aug 04 00:36:33.879117 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:36:33.879026 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd\": container with ID starting with 0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd not found: ID does not exist" containerID="0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd" Aug 04 00:36:33.879117 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.879056 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd"} err="failed to get container status \"0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd\": rpc error: code = NotFound desc = could not find container \"0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd\": container with ID starting with 0bcd4cb5773b8c816e76b0bf8aca53453e521c752dcb005a0b04913240d517fd not found: ID does not exist" Aug 04 00:36:33.881581 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.881558 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z"] Aug 04 00:36:33.889291 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.889268 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ae02d212-gw-openshift-default-56449745c6-lkw7z"] Aug 04 00:36:33.953537 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:33.953491 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" path="/var/lib/kubelet/pods/d78037fc-b597-4b3c-a893-c9b404767ff6/volumes" Aug 04 00:36:44.926180 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:44.926144 2552 generic.go:356] "Generic (PLEG): container finished" podID="f25dad10-72e1-4baf-847d-a49815c60216" containerID="af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6" exitCode=0 Aug 04 00:36:44.926584 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:44.926220 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" event={"ID":"f25dad10-72e1-4baf-847d-a49815c60216","Type":"ContainerDied","Data":"af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6"} Aug 04 00:36:44.926627 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:44.926602 2552 scope.go:117] "RemoveContainer" containerID="af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6" Aug 04 00:36:44.927685 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:44.927658 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:36:44.927749 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:44.927724 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:36:44.927784 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:44.927761 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:36:45.932635 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:45.932594 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" event={"ID":"f25dad10-72e1-4baf-847d-a49815c60216","Type":"ContainerStarted","Data":"7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d"} Aug 04 00:36:45.949733 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:45.949704 2552 patch_prober.go:28] interesting pod/e2e-ait-62613146-openshift-default-5cd7f87474-tt858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 04 00:36:45.949892 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:45.949755 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 04 00:36:45.952984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:45.952955 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:46.849496 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:46.849467 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:46.948094 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:46.948060 2552 patch_prober.go:28] interesting pod/e2e-ait-62613146-openshift-default-5cd7f87474-tt858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 04 00:36:46.948543 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:46.948117 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 04 00:36:47.948042 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:47.948005 2552 patch_prober.go:28] interesting pod/e2e-ait-62613146-openshift-default-5cd7f87474-tt858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 04 00:36:47.948214 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:47.948071 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 04 00:36:48.948597 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:48.948561 2552 patch_prober.go:28] interesting pod/e2e-ait-62613146-openshift-default-5cd7f87474-tt858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 04 00:36:48.949036 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:48.948612 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 04 00:36:49.948041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:49.948004 2552 patch_prober.go:28] interesting pod/e2e-ait-62613146-openshift-default-5cd7f87474-tt858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 04 00:36:49.948248 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:49.948057 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 04 00:36:50.948527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:50.948492 2552 patch_prober.go:28] interesting pod/e2e-ait-62613146-openshift-default-5cd7f87474-tt858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 04 00:36:50.948971 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:50.948552 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 04 00:36:51.954020 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:51.953993 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:36:51.954940 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:36:51.954921 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:37:02.051944 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.051906 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k"] Aug 04 00:37:02.052430 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.052263 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" Aug 04 00:37:02.052430 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.052275 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" Aug 04 00:37:02.052430 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.052289 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" Aug 04 00:37:02.052430 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.052295 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" Aug 04 00:37:02.052430 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.052355 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" Aug 04 00:37:02.052430 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.052367 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78037fc-b597-4b3c-a893-c9b404767ff6" containerName="istio-proxy" Aug 04 00:37:02.055414 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.055395 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" Aug 04 00:37:02.058033 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.058009 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 00:37:02.058312 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.058284 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:37:02.061805 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.061782 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k"] Aug 04 00:37:02.128532 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.128503 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f981b54-f98d-45b9-8393-f7bade3c349a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-62613146-4f48k\" (UID: \"6f981b54-f98d-45b9-8393-f7bade3c349a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" Aug 04 00:37:02.230091 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.230051 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f981b54-f98d-45b9-8393-f7bade3c349a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-62613146-4f48k\" (UID: \"6f981b54-f98d-45b9-8393-f7bade3c349a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" Aug 04 00:37:02.230677 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.230658 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f981b54-f98d-45b9-8393-f7bade3c349a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-62613146-4f48k\" (UID: \"6f981b54-f98d-45b9-8393-f7bade3c349a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" Aug 04 00:37:02.368533 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.368504 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" Aug 04 00:37:02.498277 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:02.498252 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k"] Aug 04 00:37:03.012386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:03.012291 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" event={"ID":"6f981b54-f98d-45b9-8393-f7bade3c349a","Type":"ContainerStarted","Data":"50261c6d4861897be0ccc9fd504ffac645c2632ba5b99cdaabd0d06873237691"} Aug 04 00:37:03.012386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:03.012380 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" event={"ID":"6f981b54-f98d-45b9-8393-f7bade3c349a","Type":"ContainerStarted","Data":"d24b79973f49cfa8a33a2257d857e48deb523cbd6d1e04e0afb4df4b7070a371"} Aug 04 00:37:03.028794 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:03.028744 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" podStartSLOduration=1.028728936 podStartE2EDuration="1.028728936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:37:03.026809836 +0000 UTC m=+2061.689719318" watchObservedRunningTime="2026-08-04 00:37:03.028728936 +0000 UTC m=+2061.691638413" Aug 04 00:37:04.018170 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:04.018130 2552 generic.go:356] "Generic (PLEG): container finished" podID="6f981b54-f98d-45b9-8393-f7bade3c349a" containerID="50261c6d4861897be0ccc9fd504ffac645c2632ba5b99cdaabd0d06873237691" exitCode=0 Aug 04 00:37:04.018562 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:04.018207 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" event={"ID":"6f981b54-f98d-45b9-8393-f7bade3c349a","Type":"ContainerDied","Data":"50261c6d4861897be0ccc9fd504ffac645c2632ba5b99cdaabd0d06873237691"} Aug 04 00:37:05.166331 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:05.166309 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" Aug 04 00:37:05.259645 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:05.259611 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f981b54-f98d-45b9-8393-f7bade3c349a-maas-api-service-ca\") pod \"6f981b54-f98d-45b9-8393-f7bade3c349a\" (UID: \"6f981b54-f98d-45b9-8393-f7bade3c349a\") " Aug 04 00:37:05.260002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:05.259974 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f981b54-f98d-45b9-8393-f7bade3c349a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f981b54-f98d-45b9-8393-f7bade3c349a" (UID: "6f981b54-f98d-45b9-8393-f7bade3c349a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:05.360665 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:05.360584 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f981b54-f98d-45b9-8393-f7bade3c349a-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:06.029339 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:06.029306 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" event={"ID":"6f981b54-f98d-45b9-8393-f7bade3c349a","Type":"ContainerDied","Data":"d24b79973f49cfa8a33a2257d857e48deb523cbd6d1e04e0afb4df4b7070a371"} Aug 04 00:37:06.029484 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:06.029358 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d24b79973f49cfa8a33a2257d857e48deb523cbd6d1e04e0afb4df4b7070a371" Aug 04 00:37:06.029484 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:06.029319 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k" Aug 04 00:37:12.977989 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:12.977954 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k"] Aug 04 00:37:12.980528 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:12.980502 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62613146-4f48k"] Aug 04 00:37:13.953944 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:13.953905 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f981b54-f98d-45b9-8393-f7bade3c349a" path="/var/lib/kubelet/pods/6f981b54-f98d-45b9-8393-f7bade3c349a/volumes" Aug 04 00:37:14.114140 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.114110 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858"] Aug 04 00:37:14.114507 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.114331 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" containerID="cri-o://7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d" gracePeriod=30 Aug 04 00:37:14.829329 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.829295 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7"] Aug 04 00:37:14.829710 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.829694 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f981b54-f98d-45b9-8393-f7bade3c349a" containerName="revoke-keys" Aug 04 00:37:14.829710 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.829711 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f981b54-f98d-45b9-8393-f7bade3c349a" containerName="revoke-keys" Aug 04 00:37:14.829829 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.829781 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f981b54-f98d-45b9-8393-f7bade3c349a" containerName="revoke-keys" Aug 04 00:37:14.832941 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.832926 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:14.836916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.836894 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fe5216a4-openshift-default-dockercfg-tcmbv\"" Aug 04 00:37:14.845195 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.845175 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7"] Aug 04 00:37:14.947677 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.947646 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-data\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:14.947852 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.947690 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-socket\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:14.947852 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.947790 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-credential-socket\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:14.947852 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.947840 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6947846-ede6-41cd-9065-00d461599aa2-istio-podinfo\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:14.948022 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.947933 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6947846-ede6-41cd-9065-00d461599aa2-istiod-ca-cert\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:14.948022 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.947950 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7n4\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-kube-api-access-xw7n4\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:14.948022 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.947977 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-envoy\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:14.948022 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.948001 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-certs\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:14.948022 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:14.948019 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-istio-token\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.048654 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.048619 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6947846-ede6-41cd-9065-00d461599aa2-istio-podinfo\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.048819 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.048680 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6947846-ede6-41cd-9065-00d461599aa2-istiod-ca-cert\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.048819 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.048700 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7n4\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-kube-api-access-xw7n4\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.048819 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.048726 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-envoy\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.048819 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.048748 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-certs\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.048819 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.048763 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-istio-token\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.048819 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.048779 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-data\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.048819 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.048806 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-socket\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.049171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.048884 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-credential-socket\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.049469 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.049438 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-certs\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.049603 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.049467 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-data\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.049603 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.049448 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6947846-ede6-41cd-9065-00d461599aa2-istiod-ca-cert\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.049603 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.049506 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-credential-socket\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.049603 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.049595 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-socket\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.051280 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.051258 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-envoy\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.051498 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.051480 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6947846-ede6-41cd-9065-00d461599aa2-istio-podinfo\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.057109 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.057085 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-istio-token\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.057328 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.057312 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7n4\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-kube-api-access-xw7n4\") pod \"e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.143300 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.143271 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:15.281071 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.281044 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7"] Aug 04 00:37:15.283211 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:37:15.283184 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6947846_ede6_41cd_9065_00d461599aa2.slice/crio-acc7a5c74fe4c4e864c677d56d78d595c5916db14521fec352279da63a41f391 WatchSource:0}: Error finding container acc7a5c74fe4c4e864c677d56d78d595c5916db14521fec352279da63a41f391: Status 404 returned error can't find the container with id acc7a5c74fe4c4e864c677d56d78d595c5916db14521fec352279da63a41f391 Aug 04 00:37:15.285921 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.285834 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:37:15.286010 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.285970 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:37:15.286051 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:15.286018 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:37:16.081081 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:16.081048 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" event={"ID":"f6947846-ede6-41cd-9065-00d461599aa2","Type":"ContainerStarted","Data":"ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be"} Aug 04 00:37:16.081275 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:16.081099 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" event={"ID":"f6947846-ede6-41cd-9065-00d461599aa2","Type":"ContainerStarted","Data":"acc7a5c74fe4c4e864c677d56d78d595c5916db14521fec352279da63a41f391"} Aug 04 00:37:16.102168 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:16.102120 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" podStartSLOduration=2.102103153 podStartE2EDuration="2.102103153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:37:16.101394385 +0000 UTC m=+2074.764303864" watchObservedRunningTime="2026-08-04 00:37:16.102103153 +0000 UTC m=+2074.765012709" Aug 04 00:37:16.143844 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:16.143806 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:16.148639 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:16.148617 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:17.086082 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:17.086048 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:17.086841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:17.086819 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:19.370076 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.370043 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:37:19.494643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494555 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25dad10-72e1-4baf-847d-a49815c60216-istiod-ca-cert\") pod \"f25dad10-72e1-4baf-847d-a49815c60216\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " Aug 04 00:37:19.494643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494600 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-istio-token\") pod \"f25dad10-72e1-4baf-847d-a49815c60216\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " Aug 04 00:37:19.494643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494626 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-credential-socket\") pod \"f25dad10-72e1-4baf-847d-a49815c60216\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " Aug 04 00:37:19.494956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494673 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-data\") pod \"f25dad10-72e1-4baf-847d-a49815c60216\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " Aug 04 00:37:19.494956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494716 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25dad10-72e1-4baf-847d-a49815c60216-istio-podinfo\") pod \"f25dad10-72e1-4baf-847d-a49815c60216\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " Aug 04 00:37:19.494956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494752 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-envoy\") pod \"f25dad10-72e1-4baf-847d-a49815c60216\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " Aug 04 00:37:19.494956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494781 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-certs\") pod \"f25dad10-72e1-4baf-847d-a49815c60216\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " Aug 04 00:37:19.494956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494897 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-socket\") pod \"f25dad10-72e1-4baf-847d-a49815c60216\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " Aug 04 00:37:19.494956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494943 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25dad10-72e1-4baf-847d-a49815c60216-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f25dad10-72e1-4baf-847d-a49815c60216" (UID: "f25dad10-72e1-4baf-847d-a49815c60216"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:19.495246 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.494954 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq2l6\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-kube-api-access-bq2l6\") pod \"f25dad10-72e1-4baf-847d-a49815c60216\" (UID: \"f25dad10-72e1-4baf-847d-a49815c60216\") " Aug 04 00:37:19.495246 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.495012 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-data" (OuterVolumeSpecName: "istio-data") pod "f25dad10-72e1-4baf-847d-a49815c60216" (UID: "f25dad10-72e1-4baf-847d-a49815c60216"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:19.495246 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.495203 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f25dad10-72e1-4baf-847d-a49815c60216" (UID: "f25dad10-72e1-4baf-847d-a49815c60216"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:19.495394 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.495346 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:19.495394 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.495366 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:19.495394 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.495383 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25dad10-72e1-4baf-847d-a49815c60216-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:19.495668 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.495622 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f25dad10-72e1-4baf-847d-a49815c60216" (UID: "f25dad10-72e1-4baf-847d-a49815c60216"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:19.495775 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.495659 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f25dad10-72e1-4baf-847d-a49815c60216" (UID: "f25dad10-72e1-4baf-847d-a49815c60216"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:19.497466 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.497444 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-istio-token" (OuterVolumeSpecName: "istio-token") pod "f25dad10-72e1-4baf-847d-a49815c60216" (UID: "f25dad10-72e1-4baf-847d-a49815c60216"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:19.497547 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.497474 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-kube-api-access-bq2l6" (OuterVolumeSpecName: "kube-api-access-bq2l6") pod "f25dad10-72e1-4baf-847d-a49815c60216" (UID: "f25dad10-72e1-4baf-847d-a49815c60216"). InnerVolumeSpecName "kube-api-access-bq2l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:19.497547 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.497474 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f25dad10-72e1-4baf-847d-a49815c60216-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f25dad10-72e1-4baf-847d-a49815c60216" (UID: "f25dad10-72e1-4baf-847d-a49815c60216"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:37:19.497547 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.497474 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f25dad10-72e1-4baf-847d-a49815c60216" (UID: "f25dad10-72e1-4baf-847d-a49815c60216"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:19.596336 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.596305 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25dad10-72e1-4baf-847d-a49815c60216-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:19.596336 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.596330 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:19.596336 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.596340 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:19.596550 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.596348 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq2l6\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-kube-api-access-bq2l6\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:19.596550 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.596357 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25dad10-72e1-4baf-847d-a49815c60216-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:19.596550 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:19.596366 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25dad10-72e1-4baf-847d-a49815c60216-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:20.102170 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.102129 2552 generic.go:356] "Generic (PLEG): container finished" podID="f25dad10-72e1-4baf-847d-a49815c60216" containerID="7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d" exitCode=0 Aug 04 00:37:20.102362 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.102187 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" Aug 04 00:37:20.102362 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.102217 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" event={"ID":"f25dad10-72e1-4baf-847d-a49815c60216","Type":"ContainerDied","Data":"7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d"} Aug 04 00:37:20.102362 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.102284 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858" event={"ID":"f25dad10-72e1-4baf-847d-a49815c60216","Type":"ContainerDied","Data":"0710c51a942cfe13c7d6765e6b30d93b94f66165e2686f9b87b44947b2a586a6"} Aug 04 00:37:20.102362 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.102314 2552 scope.go:117] "RemoveContainer" containerID="7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d" Aug 04 00:37:20.111102 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.111084 2552 scope.go:117] "RemoveContainer" containerID="af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6" Aug 04 00:37:20.119625 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.119606 2552 scope.go:117] "RemoveContainer" containerID="7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d" Aug 04 00:37:20.119914 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:37:20.119892 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d\": container with ID starting with 7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d not found: ID does not exist" containerID="7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d" Aug 04 00:37:20.120004 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.119926 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d"} err="failed to get container status \"7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d\": rpc error: code = NotFound desc = could not find container \"7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d\": container with ID starting with 7b88a9dcf5965bc310e821c10f28eaf71800c0756d5b64d7f8897f2be8f6199d not found: ID does not exist" Aug 04 00:37:20.120004 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.119952 2552 scope.go:117] "RemoveContainer" containerID="af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6" Aug 04 00:37:20.120228 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:37:20.120211 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6\": container with ID starting with af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6 not found: ID does not exist" containerID="af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6" Aug 04 00:37:20.120272 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.120235 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6"} err="failed to get container status \"af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6\": rpc error: code = NotFound desc = could not find container \"af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6\": container with ID starting with af75823337bbe09ab1e57e35f1710c4af33a14dbe218bb7065b389d873ae5ed6 not found: ID does not exist" Aug 04 00:37:20.120851 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.120834 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858"] Aug 04 00:37:20.124299 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:20.124273 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-62613146-openshift-default-5cd7f87474-tt858"] Aug 04 00:37:21.953265 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:21.953234 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f25dad10-72e1-4baf-847d-a49815c60216" path="/var/lib/kubelet/pods/f25dad10-72e1-4baf-847d-a49815c60216/volumes" Aug 04 00:37:30.158128 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:30.158089 2552 generic.go:356] "Generic (PLEG): container finished" podID="f6947846-ede6-41cd-9065-00d461599aa2" containerID="ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be" exitCode=0 Aug 04 00:37:30.158590 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:30.158172 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" event={"ID":"f6947846-ede6-41cd-9065-00d461599aa2","Type":"ContainerDied","Data":"ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be"} Aug 04 00:37:30.158660 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:30.158641 2552 scope.go:117] "RemoveContainer" containerID="ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be" Aug 04 00:37:30.159919 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:30.159894 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:37:30.160013 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:30.159957 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:37:30.160013 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:30.160004 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:37:31.173891 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:31.173390 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" event={"ID":"f6947846-ede6-41cd-9065-00d461599aa2","Type":"ContainerStarted","Data":"87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071"} Aug 04 00:37:32.085992 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:32.085952 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:32.144078 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:32.144045 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:32.145037 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:32.145012 2552 patch_prober.go:28] interesting pod/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 00:37:32.145146 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:32.145073 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 00:37:33.144684 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:33.144646 2552 patch_prober.go:28] interesting pod/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 00:37:33.145092 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:33.144709 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 00:37:34.144285 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:34.144244 2552 patch_prober.go:28] interesting pod/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 00:37:34.144542 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:34.144314 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 00:37:35.143961 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:35.143926 2552 patch_prober.go:28] interesting pod/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 00:37:35.144418 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:35.143990 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 00:37:36.144604 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:36.144564 2552 patch_prober.go:28] interesting pod/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 00:37:36.145032 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:36.144634 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 00:37:37.148186 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:37.148157 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:37.149143 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:37.149124 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:42.302095 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.302017 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z"] Aug 04 00:37:42.302442 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.302347 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" Aug 04 00:37:42.302442 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.302358 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" Aug 04 00:37:42.302442 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.302431 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" Aug 04 00:37:42.302442 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.302438 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" Aug 04 00:37:42.306802 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.306784 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" Aug 04 00:37:42.310294 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.310170 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 00:37:42.310294 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.310170 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:37:42.311768 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.311745 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z"] Aug 04 00:37:42.399428 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.399397 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc51e080-b847-4256-b3af-a209888d2c33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z\" (UID: \"bc51e080-b847-4256-b3af-a209888d2c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" Aug 04 00:37:42.500226 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.500191 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc51e080-b847-4256-b3af-a209888d2c33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z\" (UID: \"bc51e080-b847-4256-b3af-a209888d2c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" Aug 04 00:37:42.500831 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.500810 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc51e080-b847-4256-b3af-a209888d2c33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z\" (UID: \"bc51e080-b847-4256-b3af-a209888d2c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" Aug 04 00:37:42.618728 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.618683 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" Aug 04 00:37:42.748427 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:42.748370 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z"] Aug 04 00:37:42.750683 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:37:42.750652 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc51e080_b847_4256_b3af_a209888d2c33.slice/crio-607a2a30cd7b4bb67f6973d07c2411e308f2d67586238668f3f2f049407fe386 WatchSource:0}: Error finding container 607a2a30cd7b4bb67f6973d07c2411e308f2d67586238668f3f2f049407fe386: Status 404 returned error can't find the container with id 607a2a30cd7b4bb67f6973d07c2411e308f2d67586238668f3f2f049407fe386 Aug 04 00:37:43.243989 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:43.243900 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" event={"ID":"bc51e080-b847-4256-b3af-a209888d2c33","Type":"ContainerStarted","Data":"1d6748b69401e9b736e04c7e4692688aa7801fc1d2b43f4b9b533c875debdfd3"} Aug 04 00:37:43.243989 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:43.243966 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" event={"ID":"bc51e080-b847-4256-b3af-a209888d2c33","Type":"ContainerStarted","Data":"607a2a30cd7b4bb67f6973d07c2411e308f2d67586238668f3f2f049407fe386"} Aug 04 00:37:43.262710 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:43.262659 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" podStartSLOduration=1.26264361 podStartE2EDuration="1.26264361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:37:43.261281346 +0000 UTC m=+2101.924190826" watchObservedRunningTime="2026-08-04 00:37:43.26264361 +0000 UTC m=+2101.925553088" Aug 04 00:37:44.250734 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:44.250702 2552 generic.go:356] "Generic (PLEG): container finished" podID="bc51e080-b847-4256-b3af-a209888d2c33" containerID="1d6748b69401e9b736e04c7e4692688aa7801fc1d2b43f4b9b533c875debdfd3" exitCode=0 Aug 04 00:37:44.251124 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:44.250779 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" event={"ID":"bc51e080-b847-4256-b3af-a209888d2c33","Type":"ContainerDied","Data":"1d6748b69401e9b736e04c7e4692688aa7801fc1d2b43f4b9b533c875debdfd3"} Aug 04 00:37:45.387310 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:45.387286 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" Aug 04 00:37:45.526042 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:45.525958 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc51e080-b847-4256-b3af-a209888d2c33-maas-api-service-ca\") pod \"bc51e080-b847-4256-b3af-a209888d2c33\" (UID: \"bc51e080-b847-4256-b3af-a209888d2c33\") " Aug 04 00:37:45.526325 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:45.526301 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc51e080-b847-4256-b3af-a209888d2c33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc51e080-b847-4256-b3af-a209888d2c33" (UID: "bc51e080-b847-4256-b3af-a209888d2c33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:45.627408 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:45.627374 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc51e080-b847-4256-b3af-a209888d2c33-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:46.261800 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:46.261718 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" Aug 04 00:37:46.261984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:46.261717 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z" event={"ID":"bc51e080-b847-4256-b3af-a209888d2c33","Type":"ContainerDied","Data":"607a2a30cd7b4bb67f6973d07c2411e308f2d67586238668f3f2f049407fe386"} Aug 04 00:37:46.261984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:46.261898 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="607a2a30cd7b4bb67f6973d07c2411e308f2d67586238668f3f2f049407fe386" Aug 04 00:37:49.438427 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:49.438390 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z"] Aug 04 00:37:49.440780 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:49.440756 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fe5216a4-4tt5z"] Aug 04 00:37:49.603311 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:49.603277 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7"] Aug 04 00:37:49.603596 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:49.603546 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" containerID="cri-o://87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071" gracePeriod=30 Aug 04 00:37:49.953202 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:49.953169 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc51e080-b847-4256-b3af-a209888d2c33" path="/var/lib/kubelet/pods/bc51e080-b847-4256-b3af-a209888d2c33/volumes" Aug 04 00:37:52.151034 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:52.150994 2552 patch_prober.go:28] interesting pod/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 04 00:37:52.151408 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:52.151070 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:37:54.866297 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:54.866239 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:55.014227 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014194 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-istio-token\") pod \"f6947846-ede6-41cd-9065-00d461599aa2\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " Aug 04 00:37:55.014419 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014239 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw7n4\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-kube-api-access-xw7n4\") pod \"f6947846-ede6-41cd-9065-00d461599aa2\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " Aug 04 00:37:55.014419 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014271 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6947846-ede6-41cd-9065-00d461599aa2-istio-podinfo\") pod \"f6947846-ede6-41cd-9065-00d461599aa2\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " Aug 04 00:37:55.014419 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014298 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-credential-socket\") pod \"f6947846-ede6-41cd-9065-00d461599aa2\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " Aug 04 00:37:55.014419 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014315 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6947846-ede6-41cd-9065-00d461599aa2-istiod-ca-cert\") pod \"f6947846-ede6-41cd-9065-00d461599aa2\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " Aug 04 00:37:55.014419 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014342 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-certs\") pod \"f6947846-ede6-41cd-9065-00d461599aa2\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " Aug 04 00:37:55.014419 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014393 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-socket\") pod \"f6947846-ede6-41cd-9065-00d461599aa2\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " Aug 04 00:37:55.014847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014430 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-data\") pod \"f6947846-ede6-41cd-9065-00d461599aa2\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " Aug 04 00:37:55.014847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014473 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-envoy\") pod \"f6947846-ede6-41cd-9065-00d461599aa2\" (UID: \"f6947846-ede6-41cd-9065-00d461599aa2\") " Aug 04 00:37:55.014847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014646 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f6947846-ede6-41cd-9065-00d461599aa2" (UID: "f6947846-ede6-41cd-9065-00d461599aa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:55.014847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014692 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f6947846-ede6-41cd-9065-00d461599aa2" (UID: "f6947846-ede6-41cd-9065-00d461599aa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:55.014847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014666 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f6947846-ede6-41cd-9065-00d461599aa2" (UID: "f6947846-ede6-41cd-9065-00d461599aa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:55.014847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014701 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6947846-ede6-41cd-9065-00d461599aa2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f6947846-ede6-41cd-9065-00d461599aa2" (UID: "f6947846-ede6-41cd-9065-00d461599aa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:55.014847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014787 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:55.014847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014824 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:55.014847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014841 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6947846-ede6-41cd-9065-00d461599aa2-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:55.015346 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014885 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:55.015346 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.014912 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-data" (OuterVolumeSpecName: "istio-data") pod "f6947846-ede6-41cd-9065-00d461599aa2" (UID: "f6947846-ede6-41cd-9065-00d461599aa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:55.016488 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.016468 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6947846-ede6-41cd-9065-00d461599aa2" (UID: "f6947846-ede6-41cd-9065-00d461599aa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:55.016558 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.016540 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6947846-ede6-41cd-9065-00d461599aa2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f6947846-ede6-41cd-9065-00d461599aa2" (UID: "f6947846-ede6-41cd-9065-00d461599aa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:37:55.017030 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.017007 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f6947846-ede6-41cd-9065-00d461599aa2" (UID: "f6947846-ede6-41cd-9065-00d461599aa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:37:55.017030 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.017018 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-kube-api-access-xw7n4" (OuterVolumeSpecName: "kube-api-access-xw7n4") pod "f6947846-ede6-41cd-9065-00d461599aa2" (UID: "f6947846-ede6-41cd-9065-00d461599aa2"). InnerVolumeSpecName "kube-api-access-xw7n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:55.115647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.115617 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:55.115647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.115645 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6947846-ede6-41cd-9065-00d461599aa2-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:55.115846 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.115656 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:55.115846 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.115672 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7n4\" (UniqueName: \"kubernetes.io/projected/f6947846-ede6-41cd-9065-00d461599aa2-kube-api-access-xw7n4\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:55.115846 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.115685 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6947846-ede6-41cd-9065-00d461599aa2-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:37:55.310402 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.310303 2552 generic.go:356] "Generic (PLEG): container finished" podID="f6947846-ede6-41cd-9065-00d461599aa2" containerID="87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071" exitCode=0 Aug 04 00:37:55.310402 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.310363 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" Aug 04 00:37:55.310402 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.310385 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" event={"ID":"f6947846-ede6-41cd-9065-00d461599aa2","Type":"ContainerDied","Data":"87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071"} Aug 04 00:37:55.310626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.310458 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7" event={"ID":"f6947846-ede6-41cd-9065-00d461599aa2","Type":"ContainerDied","Data":"acc7a5c74fe4c4e864c677d56d78d595c5916db14521fec352279da63a41f391"} Aug 04 00:37:55.310626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.310500 2552 scope.go:117] "RemoveContainer" containerID="87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071" Aug 04 00:37:55.320657 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.320635 2552 scope.go:117] "RemoveContainer" containerID="ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be" Aug 04 00:37:55.328956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.328933 2552 scope.go:117] "RemoveContainer" containerID="87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071" Aug 04 00:37:55.329219 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:37:55.329201 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071\": container with ID starting with 87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071 not found: ID does not exist" containerID="87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071" Aug 04 00:37:55.329282 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.329229 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071"} err="failed to get container status \"87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071\": rpc error: code = NotFound desc = could not find container \"87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071\": container with ID starting with 87855ffb29bd807d77670decc5f5e81d53c870c9f892981a39173ef7cf852071 not found: ID does not exist" Aug 04 00:37:55.329282 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.329245 2552 scope.go:117] "RemoveContainer" containerID="ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be" Aug 04 00:37:55.329480 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:37:55.329460 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be\": container with ID starting with ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be not found: ID does not exist" containerID="ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be" Aug 04 00:37:55.329551 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.329483 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be"} err="failed to get container status \"ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be\": rpc error: code = NotFound desc = could not find container \"ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be\": container with ID starting with ff702113fc0b89d9303c6492e35ca41d08ff49b6709a89b9df28775da819d5be not found: ID does not exist" Aug 04 00:37:55.331909 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.331887 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7"] Aug 04 00:37:55.335586 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.335567 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fe5216a4-openshift-default-69d474dd6d-n7wr7"] Aug 04 00:37:55.953119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:37:55.953090 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6947846-ede6-41cd-9065-00d461599aa2" path="/var/lib/kubelet/pods/f6947846-ede6-41cd-9065-00d461599aa2/volumes" Aug 04 00:39:46.042291 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042253 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27"] Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042610 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc51e080-b847-4256-b3af-a209888d2c33" containerName="revoke-keys" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042625 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc51e080-b847-4256-b3af-a209888d2c33" containerName="revoke-keys" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042641 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042650 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042659 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042664 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042677 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042682 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25dad10-72e1-4baf-847d-a49815c60216" containerName="istio-proxy" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042736 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042746 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc51e080-b847-4256-b3af-a209888d2c33" containerName="revoke-keys" Aug 04 00:39:46.042777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.042754 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6947846-ede6-41cd-9065-00d461599aa2" containerName="istio-proxy" Aug 04 00:39:46.045671 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.045656 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.048268 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.048246 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4c743254-openshift-default-dockercfg-f7mrb\"" Aug 04 00:39:46.068207 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.068182 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27"] Aug 04 00:39:46.184908 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.184842 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-data\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.185096 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.184991 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxfq9\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-kube-api-access-xxfq9\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.185096 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.185028 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-socket\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.185096 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.185045 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-credential-socket\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.185096 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.185060 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-podinfo\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.185294 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.185102 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istiod-ca-cert\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.185294 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.185156 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-envoy\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.185294 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.185184 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-certs\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.185294 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.185210 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-token\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.286341 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286312 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxfq9\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-kube-api-access-xxfq9\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.286532 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286363 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-socket\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.286532 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286380 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-credential-socket\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.286532 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286396 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-podinfo\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.286532 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286420 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istiod-ca-cert\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.286751 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286535 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-envoy\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.286751 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286574 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-certs\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.286751 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286603 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-token\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.286751 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286674 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-data\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.287001 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.286776 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-socket\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.287062 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.287022 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-certs\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.287119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.287056 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-data\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.287119 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.287107 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-credential-socket\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.287221 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.287137 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istiod-ca-cert\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.289291 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.289269 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-envoy\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.289413 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.289396 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-podinfo\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.298901 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.298835 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxfq9\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-kube-api-access-xxfq9\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.301601 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.301580 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-token\") pod \"e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.356273 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.356239 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:46.495370 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.495345 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27"] Aug 04 00:39:46.497085 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:39:46.497060 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba220e2_21dc_4b5f_a783_8d2cd50c2fdb.slice/crio-a4687e5e9c448a8bf0052f1a5a351c5bc4af0ad958439458143910441596ab57 WatchSource:0}: Error finding container a4687e5e9c448a8bf0052f1a5a351c5bc4af0ad958439458143910441596ab57: Status 404 returned error can't find the container with id a4687e5e9c448a8bf0052f1a5a351c5bc4af0ad958439458143910441596ab57 Aug 04 00:39:46.499228 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.499197 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:39:46.499307 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.499264 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:39:46.499307 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.499297 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:39:46.859190 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.859098 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" event={"ID":"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb","Type":"ContainerStarted","Data":"a7718240b66a4f28149ecd099718be57307348fe2b1ea6f5975e670e03eb5587"} Aug 04 00:39:46.859190 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.859152 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" event={"ID":"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb","Type":"ContainerStarted","Data":"a4687e5e9c448a8bf0052f1a5a351c5bc4af0ad958439458143910441596ab57"} Aug 04 00:39:46.880289 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:46.880238 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" podStartSLOduration=0.880221846 podStartE2EDuration="880.221846ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:39:46.878758453 +0000 UTC m=+2225.541667958" watchObservedRunningTime="2026-08-04 00:39:46.880221846 +0000 UTC m=+2225.543131323" Aug 04 00:39:47.356908 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:47.356872 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:47.361936 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:47.361910 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:47.863738 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:47.863705 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:47.864709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:47.864687 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:39:50.489497 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.489465 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk"] Aug 04 00:39:50.492977 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.492954 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:39:50.495328 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.495306 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4c743254\"" Aug 04 00:39:50.503178 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.503144 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk"] Aug 04 00:39:50.521337 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.521312 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c856859-4587-4489-b5ad-ad929036a46d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk\" (UID: \"6c856859-4587-4489-b5ad-ad929036a46d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:39:50.521444 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.521353 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnz9\" (UniqueName: \"kubernetes.io/projected/6c856859-4587-4489-b5ad-ad929036a46d-kube-api-access-fwnz9\") pod \"payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk\" (UID: \"6c856859-4587-4489-b5ad-ad929036a46d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:39:50.622698 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.622660 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c856859-4587-4489-b5ad-ad929036a46d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk\" (UID: \"6c856859-4587-4489-b5ad-ad929036a46d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:39:50.622897 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.622731 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnz9\" (UniqueName: \"kubernetes.io/projected/6c856859-4587-4489-b5ad-ad929036a46d-kube-api-access-fwnz9\") pod \"payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk\" (UID: \"6c856859-4587-4489-b5ad-ad929036a46d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:39:50.623498 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.623471 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c856859-4587-4489-b5ad-ad929036a46d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk\" (UID: \"6c856859-4587-4489-b5ad-ad929036a46d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:39:50.632139 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.632109 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnz9\" (UniqueName: \"kubernetes.io/projected/6c856859-4587-4489-b5ad-ad929036a46d-kube-api-access-fwnz9\") pod \"payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk\" (UID: \"6c856859-4587-4489-b5ad-ad929036a46d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:39:50.805317 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.805233 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:39:50.937522 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:50.937501 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk"] Aug 04 00:39:50.939518 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:39:50.939494 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c856859_4587_4489_b5ad_ad929036a46d.slice/crio-bc9f7dca5eeb87bff077f75991274f7752e949fa846b300c424a0a9c0c5c9b5d WatchSource:0}: Error finding container bc9f7dca5eeb87bff077f75991274f7752e949fa846b300c424a0a9c0c5c9b5d: Status 404 returned error can't find the container with id bc9f7dca5eeb87bff077f75991274f7752e949fa846b300c424a0a9c0c5c9b5d Aug 04 00:39:51.886615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:51.886579 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" event={"ID":"6c856859-4587-4489-b5ad-ad929036a46d","Type":"ContainerStarted","Data":"d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253"} Aug 04 00:39:51.887216 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:51.886640 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" event={"ID":"6c856859-4587-4489-b5ad-ad929036a46d","Type":"ContainerStarted","Data":"bc9f7dca5eeb87bff077f75991274f7752e949fa846b300c424a0a9c0c5c9b5d"} Aug 04 00:39:51.887216 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:51.886672 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:39:51.902265 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:39:51.902205 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" podStartSLOduration=1.902190388 podStartE2EDuration="1.902190388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:39:51.901007203 +0000 UTC m=+2230.563916682" watchObservedRunningTime="2026-08-04 00:39:51.902190388 +0000 UTC m=+2230.565099866" Aug 04 00:40:02.865729 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:02.865691 2552 patch_prober.go:28] interesting pod/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 00:40:02.866150 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:02.865760 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 00:40:02.893767 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:02.893744 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:40:02.943461 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:02.943384 2552 generic.go:356] "Generic (PLEG): container finished" podID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerID="a7718240b66a4f28149ecd099718be57307348fe2b1ea6f5975e670e03eb5587" exitCode=0 Aug 04 00:40:02.943599 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:02.943453 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" event={"ID":"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb","Type":"ContainerDied","Data":"a7718240b66a4f28149ecd099718be57307348fe2b1ea6f5975e670e03eb5587"} Aug 04 00:40:02.943836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:02.943823 2552 scope.go:117] "RemoveContainer" containerID="a7718240b66a4f28149ecd099718be57307348fe2b1ea6f5975e670e03eb5587" Aug 04 00:40:02.944807 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:02.944783 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:40:02.944947 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:02.944844 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:40:02.944947 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:02.944907 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:40:03.953631 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:03.953595 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" event={"ID":"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb","Type":"ContainerStarted","Data":"2f3cc58c39f284252b8612b3d92f6392ff0378757796c830febe5c7a95a8b9d7"} Aug 04 00:40:04.357091 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:04.356997 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:40:04.357268 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:04.357233 2552 patch_prober.go:28] interesting pod/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 00:40:04.357349 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:04.357301 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 00:40:05.356946 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:05.356908 2552 patch_prober.go:28] interesting pod/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 00:40:05.357345 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:05.356965 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 00:40:06.357460 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:06.357421 2552 patch_prober.go:28] interesting pod/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 00:40:06.357910 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:06.357477 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 00:40:07.356987 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:07.356948 2552 patch_prober.go:28] interesting pod/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 00:40:07.357171 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:07.357002 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 00:40:08.356702 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:08.356663 2552 patch_prober.go:28] interesting pod/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 00:40:08.357149 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:08.356734 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 00:40:09.361141 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:09.361109 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:40:09.361584 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:09.361394 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:40:09.361993 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:09.361977 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:40:21.063157 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.063121 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg"] Aug 04 00:40:21.066522 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.066505 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" Aug 04 00:40:21.068866 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.068835 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:40:21.068967 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.068837 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 00:40:21.073228 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.073208 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg"] Aug 04 00:40:21.200829 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.200789 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d91ec0-4fd5-490a-a55a-381f115ab19e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c743254-lxndg\" (UID: \"32d91ec0-4fd5-490a-a55a-381f115ab19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" Aug 04 00:40:21.301802 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.301765 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d91ec0-4fd5-490a-a55a-381f115ab19e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c743254-lxndg\" (UID: \"32d91ec0-4fd5-490a-a55a-381f115ab19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" Aug 04 00:40:21.302461 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.302440 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d91ec0-4fd5-490a-a55a-381f115ab19e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c743254-lxndg\" (UID: \"32d91ec0-4fd5-490a-a55a-381f115ab19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" Aug 04 00:40:21.378328 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.378303 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" Aug 04 00:40:21.510462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:21.510432 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg"] Aug 04 00:40:21.511998 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:40:21.511963 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d91ec0_4fd5_490a_a55a_381f115ab19e.slice/crio-e5a29cfa2905ca122ac421279e68bd7610261c23f0ba30abfe5c10cc07c9a6c1 WatchSource:0}: Error finding container e5a29cfa2905ca122ac421279e68bd7610261c23f0ba30abfe5c10cc07c9a6c1: Status 404 returned error can't find the container with id e5a29cfa2905ca122ac421279e68bd7610261c23f0ba30abfe5c10cc07c9a6c1 Aug 04 00:40:22.045773 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:22.045676 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" event={"ID":"32d91ec0-4fd5-490a-a55a-381f115ab19e","Type":"ContainerStarted","Data":"7addc2c97104a9de1053d136c61f234fc4a1d68d64efc07be192079437cbd56a"} Aug 04 00:40:22.045773 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:22.045733 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" event={"ID":"32d91ec0-4fd5-490a-a55a-381f115ab19e","Type":"ContainerStarted","Data":"e5a29cfa2905ca122ac421279e68bd7610261c23f0ba30abfe5c10cc07c9a6c1"} Aug 04 00:40:22.060838 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:22.060782 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" podStartSLOduration=1.06076564 podStartE2EDuration="1.06076564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:40:22.059935575 +0000 UTC m=+2260.722845056" watchObservedRunningTime="2026-08-04 00:40:22.06076564 +0000 UTC m=+2260.723675117" Aug 04 00:40:23.051622 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:23.051590 2552 generic.go:356] "Generic (PLEG): container finished" podID="32d91ec0-4fd5-490a-a55a-381f115ab19e" containerID="7addc2c97104a9de1053d136c61f234fc4a1d68d64efc07be192079437cbd56a" exitCode=0 Aug 04 00:40:23.052030 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:23.051657 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" event={"ID":"32d91ec0-4fd5-490a-a55a-381f115ab19e","Type":"ContainerDied","Data":"7addc2c97104a9de1053d136c61f234fc4a1d68d64efc07be192079437cbd56a"} Aug 04 00:40:24.191358 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:24.191332 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" Aug 04 00:40:24.331835 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:24.331743 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d91ec0-4fd5-490a-a55a-381f115ab19e-maas-api-service-ca\") pod \"32d91ec0-4fd5-490a-a55a-381f115ab19e\" (UID: \"32d91ec0-4fd5-490a-a55a-381f115ab19e\") " Aug 04 00:40:24.332114 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:24.332092 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d91ec0-4fd5-490a-a55a-381f115ab19e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32d91ec0-4fd5-490a-a55a-381f115ab19e" (UID: "32d91ec0-4fd5-490a-a55a-381f115ab19e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:40:24.433449 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:24.433411 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d91ec0-4fd5-490a-a55a-381f115ab19e-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:25.063709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:25.063674 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" event={"ID":"32d91ec0-4fd5-490a-a55a-381f115ab19e","Type":"ContainerDied","Data":"e5a29cfa2905ca122ac421279e68bd7610261c23f0ba30abfe5c10cc07c9a6c1"} Aug 04 00:40:25.063709 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:25.063692 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg" Aug 04 00:40:25.063949 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:25.063724 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a29cfa2905ca122ac421279e68bd7610261c23f0ba30abfe5c10cc07c9a6c1" Aug 04 00:40:31.684924 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:31.684877 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk"] Aug 04 00:40:31.685405 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:31.685330 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" podUID="6c856859-4587-4489-b5ad-ad929036a46d" containerName="payload-pre-processing" containerID="cri-o://d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253" gracePeriod=30 Aug 04 00:40:31.841814 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:31.841780 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg"] Aug 04 00:40:31.847200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:31.846411 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c743254-lxndg"] Aug 04 00:40:31.950795 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:31.950689 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:40:31.955372 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:31.955340 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d91ec0-4fd5-490a-a55a-381f115ab19e" path="/var/lib/kubelet/pods/32d91ec0-4fd5-490a-a55a-381f115ab19e/volumes" Aug 04 00:40:32.104824 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.104790 2552 generic.go:356] "Generic (PLEG): container finished" podID="6c856859-4587-4489-b5ad-ad929036a46d" containerID="d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253" exitCode=0 Aug 04 00:40:32.105019 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.104828 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" event={"ID":"6c856859-4587-4489-b5ad-ad929036a46d","Type":"ContainerDied","Data":"d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253"} Aug 04 00:40:32.105019 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.104901 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" event={"ID":"6c856859-4587-4489-b5ad-ad929036a46d","Type":"ContainerDied","Data":"bc9f7dca5eeb87bff077f75991274f7752e949fa846b300c424a0a9c0c5c9b5d"} Aug 04 00:40:32.105019 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.104925 2552 scope.go:117] "RemoveContainer" containerID="d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253" Aug 04 00:40:32.105019 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.104837 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk" Aug 04 00:40:32.105220 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.105146 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c856859-4587-4489-b5ad-ad929036a46d-plugins-config-volume\") pod \"6c856859-4587-4489-b5ad-ad929036a46d\" (UID: \"6c856859-4587-4489-b5ad-ad929036a46d\") " Aug 04 00:40:32.105325 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.105306 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwnz9\" (UniqueName: \"kubernetes.io/projected/6c856859-4587-4489-b5ad-ad929036a46d-kube-api-access-fwnz9\") pod \"6c856859-4587-4489-b5ad-ad929036a46d\" (UID: \"6c856859-4587-4489-b5ad-ad929036a46d\") " Aug 04 00:40:32.105609 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.105576 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c856859-4587-4489-b5ad-ad929036a46d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c856859-4587-4489-b5ad-ad929036a46d" (UID: "6c856859-4587-4489-b5ad-ad929036a46d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:40:32.107527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.107504 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c856859-4587-4489-b5ad-ad929036a46d-kube-api-access-fwnz9" (OuterVolumeSpecName: "kube-api-access-fwnz9") pod "6c856859-4587-4489-b5ad-ad929036a46d" (UID: "6c856859-4587-4489-b5ad-ad929036a46d"). InnerVolumeSpecName "kube-api-access-fwnz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:40:32.122174 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.122151 2552 scope.go:117] "RemoveContainer" containerID="d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253" Aug 04 00:40:32.122450 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:40:32.122429 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253\": container with ID starting with d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253 not found: ID does not exist" containerID="d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253" Aug 04 00:40:32.122536 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.122457 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253"} err="failed to get container status \"d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253\": rpc error: code = NotFound desc = could not find container \"d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253\": container with ID starting with d677032f9c1716da5d5d54119bf53936954ab4b3cb88d1a01d6eed16246a8253 not found: ID does not exist" Aug 04 00:40:32.206220 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.206144 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwnz9\" (UniqueName: \"kubernetes.io/projected/6c856859-4587-4489-b5ad-ad929036a46d-kube-api-access-fwnz9\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:32.206220 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.206173 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c856859-4587-4489-b5ad-ad929036a46d-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:32.424724 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.424692 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk"] Aug 04 00:40:32.426648 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:32.426628 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c743254-75fc548764-fgvpk"] Aug 04 00:40:33.953056 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:33.953024 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c856859-4587-4489-b5ad-ad929036a46d" path="/var/lib/kubelet/pods/6c856859-4587-4489-b5ad-ad929036a46d/volumes" Aug 04 00:40:38.022448 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:38.022416 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27"] Aug 04 00:40:38.022917 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:38.022676 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" containerID="cri-o://2f3cc58c39f284252b8612b3d92f6392ff0378757796c830febe5c7a95a8b9d7" gracePeriod=30 Aug 04 00:40:39.363940 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:39.363905 2552 patch_prober.go:28] interesting pod/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 04 00:40:39.364317 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:39.363969 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:40:43.162186 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.162141 2552 generic.go:356] "Generic (PLEG): container finished" podID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerID="2f3cc58c39f284252b8612b3d92f6392ff0378757796c830febe5c7a95a8b9d7" exitCode=0 Aug 04 00:40:43.162580 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.162209 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" event={"ID":"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb","Type":"ContainerDied","Data":"2f3cc58c39f284252b8612b3d92f6392ff0378757796c830febe5c7a95a8b9d7"} Aug 04 00:40:43.162580 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.162273 2552 scope.go:117] "RemoveContainer" containerID="a7718240b66a4f28149ecd099718be57307348fe2b1ea6f5975e670e03eb5587" Aug 04 00:40:43.276595 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.276574 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:40:43.297594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297268 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-certs\") pod \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " Aug 04 00:40:43.297594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297336 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-envoy\") pod \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " Aug 04 00:40:43.297594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297376 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istiod-ca-cert\") pod \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " Aug 04 00:40:43.297594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297399 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-token\") pod \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " Aug 04 00:40:43.297594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297451 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-data\") pod \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " Aug 04 00:40:43.297594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297489 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-credential-socket\") pod \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " Aug 04 00:40:43.297594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297515 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-socket\") pod \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " Aug 04 00:40:43.297594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297550 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" (UID: "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:40:43.297594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297576 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-podinfo\") pod \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " Aug 04 00:40:43.298154 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297624 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxfq9\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-kube-api-access-xxfq9\") pod \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\" (UID: \"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb\") " Aug 04 00:40:43.298154 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297875 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" (UID: "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:40:43.298154 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297961 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:43.298154 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.297978 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:43.298154 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.298112 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" (UID: "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:40:43.298410 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.298246 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" (UID: "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:40:43.298410 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.298398 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" (UID: "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:40:43.300696 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.300661 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-kube-api-access-xxfq9" (OuterVolumeSpecName: "kube-api-access-xxfq9") pod "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" (UID: "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb"). InnerVolumeSpecName "kube-api-access-xxfq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:40:43.301164 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.301139 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" (UID: "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:40:43.301273 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.301201 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" (UID: "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:40:43.301399 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.301378 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" (UID: "bba220e2-21dc-4b5f-a783-8d2cd50c2fdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:40:43.399280 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.399248 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxfq9\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-kube-api-access-xxfq9\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:43.399280 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.399280 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:43.399464 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.399289 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:43.399464 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.399298 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:43.399464 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.399306 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:43.399464 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.399313 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:43.399464 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:43.399321 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:40:44.168144 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:44.168114 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" Aug 04 00:40:44.168576 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:44.168148 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27" event={"ID":"bba220e2-21dc-4b5f-a783-8d2cd50c2fdb","Type":"ContainerDied","Data":"a4687e5e9c448a8bf0052f1a5a351c5bc4af0ad958439458143910441596ab57"} Aug 04 00:40:44.168576 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:44.168207 2552 scope.go:117] "RemoveContainer" containerID="2f3cc58c39f284252b8612b3d92f6392ff0378757796c830febe5c7a95a8b9d7" Aug 04 00:40:44.185231 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:44.185207 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27"] Aug 04 00:40:44.189447 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:44.189423 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4c743254-openshift-default-57d6cc5c6-9gn27"] Aug 04 00:40:45.953256 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:40:45.953216 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" path="/var/lib/kubelet/pods/bba220e2-21dc-4b5f-a783-8d2cd50c2fdb/volumes" Aug 04 00:41:18.595978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:18.594336 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs"] Aug 04 00:41:18.595978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:18.594686 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" podUID="87d75ae1-f4bf-459c-ace9-cf9675ddb12c" containerName="manager" containerID="cri-o://f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3" gracePeriod=10 Aug 04 00:41:18.845777 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:18.845754 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:41:19.015376 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.015344 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8xhr\" (UniqueName: \"kubernetes.io/projected/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-kube-api-access-q8xhr\") pod \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\" (UID: \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\") " Aug 04 00:41:19.015572 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.015393 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-extensions-socket-volume\") pod \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\" (UID: \"87d75ae1-f4bf-459c-ace9-cf9675ddb12c\") " Aug 04 00:41:19.015787 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.015761 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "87d75ae1-f4bf-459c-ace9-cf9675ddb12c" (UID: "87d75ae1-f4bf-459c-ace9-cf9675ddb12c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:41:19.017406 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.017384 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-kube-api-access-q8xhr" (OuterVolumeSpecName: "kube-api-access-q8xhr") pod "87d75ae1-f4bf-459c-ace9-cf9675ddb12c" (UID: "87d75ae1-f4bf-459c-ace9-cf9675ddb12c"). InnerVolumeSpecName "kube-api-access-q8xhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:41:19.116356 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.116326 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8xhr\" (UniqueName: \"kubernetes.io/projected/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-kube-api-access-q8xhr\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:41:19.116356 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.116352 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87d75ae1-f4bf-459c-ace9-cf9675ddb12c-extensions-socket-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:41:19.345308 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.345220 2552 generic.go:356] "Generic (PLEG): container finished" podID="87d75ae1-f4bf-459c-ace9-cf9675ddb12c" containerID="f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3" exitCode=0 Aug 04 00:41:19.345308 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.345261 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" Aug 04 00:41:19.345524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.345305 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" event={"ID":"87d75ae1-f4bf-459c-ace9-cf9675ddb12c","Type":"ContainerDied","Data":"f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3"} Aug 04 00:41:19.345524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.345356 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs" event={"ID":"87d75ae1-f4bf-459c-ace9-cf9675ddb12c","Type":"ContainerDied","Data":"e930f0c333b0712afa9a83a1f536688fe9e9ada0d1a7c7ac62244098b6f4e1ab"} Aug 04 00:41:19.345524 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.345384 2552 scope.go:117] "RemoveContainer" containerID="f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3" Aug 04 00:41:19.355582 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.355562 2552 scope.go:117] "RemoveContainer" containerID="f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3" Aug 04 00:41:19.355830 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:41:19.355813 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3\": container with ID starting with f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3 not found: ID does not exist" containerID="f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3" Aug 04 00:41:19.355907 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.355840 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3"} err="failed to get container status \"f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3\": rpc error: code = NotFound desc = could not find container \"f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3\": container with ID starting with f86dfa74af0bb501d5f97d67613454d0a4e9c2e754b9c26cdadab0b89a2a3fd3 not found: ID does not exist" Aug 04 00:41:19.369402 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.369372 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs"] Aug 04 00:41:19.374350 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.374327 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qnmxs"] Aug 04 00:41:19.954103 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:41:19.954069 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d75ae1-f4bf-459c-ace9-cf9675ddb12c" path="/var/lib/kubelet/pods/87d75ae1-f4bf-459c-ace9-cf9675ddb12c/volumes" Aug 04 00:42:34.198640 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.198603 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7"] Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.198945 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c856859-4587-4489-b5ad-ad929036a46d" containerName="payload-pre-processing" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.198955 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c856859-4587-4489-b5ad-ad929036a46d" containerName="payload-pre-processing" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.198963 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d91ec0-4fd5-490a-a55a-381f115ab19e" containerName="revoke-keys" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.198969 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d91ec0-4fd5-490a-a55a-381f115ab19e" containerName="revoke-keys" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.198982 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.198987 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.198994 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.199000 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.199015 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d75ae1-f4bf-459c-ace9-cf9675ddb12c" containerName="manager" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.199021 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d75ae1-f4bf-459c-ace9-cf9675ddb12c" containerName="manager" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.199071 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d75ae1-f4bf-459c-ace9-cf9675ddb12c" containerName="manager" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.199081 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d91ec0-4fd5-490a-a55a-381f115ab19e" containerName="revoke-keys" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.199089 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.199095 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba220e2-21dc-4b5f-a783-8d2cd50c2fdb" containerName="istio-proxy" Aug 04 00:42:34.199304 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.199102 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c856859-4587-4489-b5ad-ad929036a46d" containerName="payload-pre-processing" Aug 04 00:42:34.202039 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.202022 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:34.204626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.204600 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tmg2q\"" Aug 04 00:42:34.220314 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.220291 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7"] Aug 04 00:42:34.274312 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.274285 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7p5\" (UniqueName: \"kubernetes.io/projected/b9e19202-0064-4b6a-9470-eeeb6094955e-kube-api-access-6h7p5\") pod \"kuadrant-operator-controller-manager-68cf589589-bs9h7\" (UID: \"b9e19202-0064-4b6a-9470-eeeb6094955e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:34.274474 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.274317 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9e19202-0064-4b6a-9470-eeeb6094955e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-bs9h7\" (UID: \"b9e19202-0064-4b6a-9470-eeeb6094955e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:34.374955 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.374919 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7p5\" (UniqueName: \"kubernetes.io/projected/b9e19202-0064-4b6a-9470-eeeb6094955e-kube-api-access-6h7p5\") pod \"kuadrant-operator-controller-manager-68cf589589-bs9h7\" (UID: \"b9e19202-0064-4b6a-9470-eeeb6094955e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:34.374955 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.374955 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9e19202-0064-4b6a-9470-eeeb6094955e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-bs9h7\" (UID: \"b9e19202-0064-4b6a-9470-eeeb6094955e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:34.375352 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.375336 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9e19202-0064-4b6a-9470-eeeb6094955e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-bs9h7\" (UID: \"b9e19202-0064-4b6a-9470-eeeb6094955e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:34.385536 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.385503 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7p5\" (UniqueName: \"kubernetes.io/projected/b9e19202-0064-4b6a-9470-eeeb6094955e-kube-api-access-6h7p5\") pod \"kuadrant-operator-controller-manager-68cf589589-bs9h7\" (UID: \"b9e19202-0064-4b6a-9470-eeeb6094955e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:34.511908 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.511799 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:34.651544 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.651511 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7"] Aug 04 00:42:34.654908 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:42:34.654882 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e19202_0064_4b6a_9470_eeeb6094955e.slice/crio-5f648555e86b67c01eb5997de4254cc0e5d3fc0eaab2ef6a5d9da4a895354aac WatchSource:0}: Error finding container 5f648555e86b67c01eb5997de4254cc0e5d3fc0eaab2ef6a5d9da4a895354aac: Status 404 returned error can't find the container with id 5f648555e86b67c01eb5997de4254cc0e5d3fc0eaab2ef6a5d9da4a895354aac Aug 04 00:42:34.718764 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:34.718736 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" event={"ID":"b9e19202-0064-4b6a-9470-eeeb6094955e","Type":"ContainerStarted","Data":"5f648555e86b67c01eb5997de4254cc0e5d3fc0eaab2ef6a5d9da4a895354aac"} Aug 04 00:42:35.725256 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:35.725219 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" event={"ID":"b9e19202-0064-4b6a-9470-eeeb6094955e","Type":"ContainerStarted","Data":"755dd62bd4b30b07e62d61670729cff7dad97b5f1e82ec58b749024bbfc4aabd"} Aug 04 00:42:35.725743 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:35.725340 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:35.747217 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:35.747172 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" podStartSLOduration=1.747158411 podStartE2EDuration="1.747158411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:42:35.744352488 +0000 UTC m=+2394.407261965" watchObservedRunningTime="2026-08-04 00:42:35.747158411 +0000 UTC m=+2394.410067888" Aug 04 00:42:42.099840 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:42.099810 2552 scope.go:117] "RemoveContainer" containerID="86d4849323761bb39100fb162ba9cf18f96f8f44f56d882e06226db295be09c3" Aug 04 00:42:42.107881 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:42.107841 2552 scope.go:117] "RemoveContainer" containerID="bb93660cc8003ee1ae04c54a26b1ecdade30f9fa443cdf38e791b9ebad555bef" Aug 04 00:42:46.733150 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:46.733114 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bs9h7" Aug 04 00:42:53.478967 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.478936 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p"] Aug 04 00:42:53.482370 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.482346 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.484903 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.484881 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-746fe6d0-openshift-default-dockercfg-k98wx\"" Aug 04 00:42:53.495023 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.495003 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p"] Aug 04 00:42:53.542101 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.542072 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-credential-socket\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.542247 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.542104 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvllf\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-kube-api-access-cvllf\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.542247 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.542124 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-podinfo\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.542247 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.542172 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istiod-ca-cert\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.542247 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.542212 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-envoy\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.542388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.542251 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-socket\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.542388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.542272 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-certs\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.542388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.542336 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-data\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.542388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.542363 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-token\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643385 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643350 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-credential-socket\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643385 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643389 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvllf\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-kube-api-access-cvllf\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643618 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643409 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-podinfo\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643618 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643424 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istiod-ca-cert\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643618 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643451 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-envoy\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643618 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643484 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-socket\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643618 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643509 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-certs\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643618 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643563 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-data\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643618 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643588 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-token\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643955 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643818 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-credential-socket\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.643955 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.643913 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-certs\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.644064 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.644024 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-socket\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.644174 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.644155 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-data\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.644358 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.644334 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istiod-ca-cert\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.645983 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.645959 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-envoy\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.646130 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.646113 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-podinfo\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.651183 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.651159 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-token\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.651283 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.651239 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvllf\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-kube-api-access-cvllf\") pod \"e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.792565 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.792486 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:53.929735 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.929709 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p"] Aug 04 00:42:53.931444 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:42:53.931412 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdabc39_32fb_4a74_a6cd_d032d896e9b0.slice/crio-5787ba0323fe64f8209e6830bd4ac5a084db2694a3e30ceb21a109705417c9c7 WatchSource:0}: Error finding container 5787ba0323fe64f8209e6830bd4ac5a084db2694a3e30ceb21a109705417c9c7: Status 404 returned error can't find the container with id 5787ba0323fe64f8209e6830bd4ac5a084db2694a3e30ceb21a109705417c9c7 Aug 04 00:42:53.933703 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.933674 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:42:53.933763 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.933734 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:42:53.933800 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:53.933764 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:42:54.819582 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:54.819542 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" event={"ID":"1cdabc39-32fb-4a74-a6cd-d032d896e9b0","Type":"ContainerStarted","Data":"4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030"} Aug 04 00:42:54.819988 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:54.819604 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" event={"ID":"1cdabc39-32fb-4a74-a6cd-d032d896e9b0","Type":"ContainerStarted","Data":"5787ba0323fe64f8209e6830bd4ac5a084db2694a3e30ceb21a109705417c9c7"} Aug 04 00:42:54.839625 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:54.839576 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" podStartSLOduration=1.8395583850000001 podStartE2EDuration="1.839558385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:42:54.838285661 +0000 UTC m=+2413.501195133" watchObservedRunningTime="2026-08-04 00:42:54.839558385 +0000 UTC m=+2413.502467864" Aug 04 00:42:55.792793 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:55.792762 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:55.797671 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:55.797644 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:55.824620 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:55.824594 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:42:55.825447 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:42:55.825429 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:43:11.357909 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:11.357876 2552 patch_prober.go:28] interesting pod/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:47588->10.133.0.57:15021: read: connection reset by peer" start-of-body= Aug 04 00:43:11.358296 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:11.357966 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:47588->10.133.0.57:15021: read: connection reset by peer" Aug 04 00:43:11.917081 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:11.917048 2552 generic.go:356] "Generic (PLEG): container finished" podID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerID="4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030" exitCode=0 Aug 04 00:43:11.917243 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:11.917119 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" event={"ID":"1cdabc39-32fb-4a74-a6cd-d032d896e9b0","Type":"ContainerDied","Data":"4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030"} Aug 04 00:43:11.917498 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:11.917483 2552 scope.go:117] "RemoveContainer" containerID="4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030" Aug 04 00:43:11.918463 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:11.918437 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:43:11.918527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:11.918508 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:43:11.918564 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:11.918548 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:43:12.924141 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:12.924103 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" event={"ID":"1cdabc39-32fb-4a74-a6cd-d032d896e9b0","Type":"ContainerStarted","Data":"aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110"} Aug 04 00:43:13.793066 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:13.793030 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:43:13.793476 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:13.793344 2552 patch_prober.go:28] interesting pod/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 00:43:13.793655 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:13.793631 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 00:43:14.793911 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:14.793845 2552 patch_prober.go:28] interesting pod/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 00:43:14.794411 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:14.793941 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 00:43:15.793490 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:15.793062 2552 patch_prober.go:28] interesting pod/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 00:43:15.793490 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:15.793124 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 00:43:16.793554 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:16.793510 2552 patch_prober.go:28] interesting pod/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 00:43:16.794092 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:16.793966 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 00:43:17.793908 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:17.793846 2552 patch_prober.go:28] interesting pod/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 00:43:17.794386 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:17.793939 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 00:43:18.796955 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:18.796924 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:43:18.797328 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:18.797191 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:43:18.797844 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:18.797829 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:43:27.722581 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.722547 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg"] Aug 04 00:43:27.727277 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.727256 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.729652 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.729625 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-746fe6d0\"/\"openshift-service-ca.crt\"" Aug 04 00:43:27.730528 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.730502 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-746fe6d0\"/\"kube-root-ca.crt\"" Aug 04 00:43:27.730641 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.730567 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-746fe6d0\"/\"e2e-lifecycle-model-746fe6d0-kserve-self-signed-certs\"" Aug 04 00:43:27.730641 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.730608 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-746fe6d0\"/\"default-dockercfg-nkz4l\"" Aug 04 00:43:27.734609 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.734580 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg"] Aug 04 00:43:27.839084 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.839046 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tmp-dir\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.839259 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.839094 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvtnv\" (UniqueName: \"kubernetes.io/projected/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-kube-api-access-kvtnv\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.839259 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.839115 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-home\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.839259 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.839140 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tls-certs\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.839259 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.839164 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-dshm\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.839259 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.839193 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-model-cache\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.939968 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.939935 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-dshm\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.940152 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.939983 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-model-cache\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.940152 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.940039 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tmp-dir\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.940152 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.940066 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtnv\" (UniqueName: \"kubernetes.io/projected/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-kube-api-access-kvtnv\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.940287 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.940154 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-home\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.940287 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.940212 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tls-certs\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.940444 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.940416 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-model-cache\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.940591 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.940486 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tmp-dir\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.940591 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.940575 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-home\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.942181 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.942154 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-dshm\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.942683 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.942662 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tls-certs\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:27.947340 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:27.947318 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtnv\" (UniqueName: \"kubernetes.io/projected/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-kube-api-access-kvtnv\") pod \"e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:28.040123 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:28.040035 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:28.172462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:28.172423 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg"] Aug 04 00:43:28.173553 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:43:28.173522 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00151e37_78f5_4ef5_bbe6_3c3f5753ce5e.slice/crio-0ea004d83b32b3c2b561b9d7d543c7fc1fe44a8cc2ad545fd92d23e88a5de16b WatchSource:0}: Error finding container 0ea004d83b32b3c2b561b9d7d543c7fc1fe44a8cc2ad545fd92d23e88a5de16b: Status 404 returned error can't find the container with id 0ea004d83b32b3c2b561b9d7d543c7fc1fe44a8cc2ad545fd92d23e88a5de16b Aug 04 00:43:29.005192 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:29.005152 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" event={"ID":"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e","Type":"ContainerStarted","Data":"f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463"} Aug 04 00:43:29.005547 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:29.005200 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" event={"ID":"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e","Type":"ContainerStarted","Data":"0ea004d83b32b3c2b561b9d7d543c7fc1fe44a8cc2ad545fd92d23e88a5de16b"} Aug 04 00:43:29.022947 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:29.022904 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" podStartSLOduration=2.022889528 podStartE2EDuration="2.022889528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:43:29.020891826 +0000 UTC m=+2447.683801305" watchObservedRunningTime="2026-08-04 00:43:29.022889528 +0000 UTC m=+2447.685799006" Aug 04 00:43:38.040918 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:38.040876 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:38.041343 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:38.040937 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:38.054189 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:38.054157 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:38.065280 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:38.065258 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:43:42.130228 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:43:42.130198 2552 scope.go:117] "RemoveContainer" containerID="50261c6d4861897be0ccc9fd504ffac645c2632ba5b99cdaabd0d06873237691" Aug 04 00:44:19.645838 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:19.645797 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg"] Aug 04 00:44:34.664993 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:34.664879 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" podUID="00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" containerName="main" containerID="cri-o://f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463" gracePeriod=45 Aug 04 00:44:34.910309 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:34.910284 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:44:35.053518 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.053428 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tmp-dir\") pod \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " Aug 04 00:44:35.053518 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.053497 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-model-cache\") pod \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " Aug 04 00:44:35.053752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.053526 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-dshm\") pod \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " Aug 04 00:44:35.053752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.053553 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvtnv\" (UniqueName: \"kubernetes.io/projected/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-kube-api-access-kvtnv\") pod \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " Aug 04 00:44:35.053752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.053628 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tls-certs\") pod \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " Aug 04 00:44:35.053752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.053661 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-home\") pod \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\" (UID: \"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e\") " Aug 04 00:44:35.053752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.053730 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" (UID: "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:44:35.053752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.053744 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" (UID: "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:44:35.054049 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.053969 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-home" (OuterVolumeSpecName: "home") pod "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" (UID: "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:44:35.054130 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.054114 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:44:35.054196 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.054134 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:44:35.054196 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.054149 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:44:35.055866 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.055831 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-kube-api-access-kvtnv" (OuterVolumeSpecName: "kube-api-access-kvtnv") pod "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" (UID: "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e"). InnerVolumeSpecName "kube-api-access-kvtnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:44:35.056153 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.056131 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-dshm" (OuterVolumeSpecName: "dshm") pod "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" (UID: "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:44:35.056199 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.056131 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" (UID: "00151e37-78f5-4ef5-bbe6-3c3f5753ce5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:44:35.155181 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.155146 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:44:35.155181 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.155177 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvtnv\" (UniqueName: \"kubernetes.io/projected/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-kube-api-access-kvtnv\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:44:35.155181 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.155188 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:44:35.329500 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.329409 2552 generic.go:356] "Generic (PLEG): container finished" podID="00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" containerID="f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463" exitCode=0 Aug 04 00:44:35.329500 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.329449 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" Aug 04 00:44:35.329714 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.329507 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" event={"ID":"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e","Type":"ContainerDied","Data":"f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463"} Aug 04 00:44:35.329714 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.329544 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg" event={"ID":"00151e37-78f5-4ef5-bbe6-3c3f5753ce5e","Type":"ContainerDied","Data":"0ea004d83b32b3c2b561b9d7d543c7fc1fe44a8cc2ad545fd92d23e88a5de16b"} Aug 04 00:44:35.329714 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.329561 2552 scope.go:117] "RemoveContainer" containerID="f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463" Aug 04 00:44:35.338815 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.338797 2552 scope.go:117] "RemoveContainer" containerID="f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463" Aug 04 00:44:35.339082 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:44:35.339064 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463\": container with ID starting with f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463 not found: ID does not exist" containerID="f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463" Aug 04 00:44:35.339145 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.339091 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463"} err="failed to get container status \"f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463\": rpc error: code = NotFound desc = could not find container \"f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463\": container with ID starting with f94600de0fa6273292b92c98f4dde3a7c36d77802276bb31141ee03a6788a463 not found: ID does not exist" Aug 04 00:44:35.350083 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.350059 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg"] Aug 04 00:44:35.355718 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.355693 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-746fe6d0/e2e-lifecycle-model-746fe6d0-kserve-58488d8c95-bjtzg"] Aug 04 00:44:35.953695 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:35.953661 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" path="/var/lib/kubelet/pods/00151e37-78f5-4ef5-bbe6-3c3f5753ce5e/volumes" Aug 04 00:44:42.148594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:44:42.148564 2552 scope.go:117] "RemoveContainer" containerID="1d6748b69401e9b736e04c7e4692688aa7801fc1d2b43f4b9b533c875debdfd3" Aug 04 00:45:02.368438 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:02.368402 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p"] Aug 04 00:45:02.368979 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:02.368638 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" containerID="cri-o://aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110" gracePeriod=30 Aug 04 00:45:03.799850 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:03.799813 2552 patch_prober.go:28] interesting pod/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 00:45:03.800253 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:03.799892 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:45:07.629132 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.629074 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:45:07.739136 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739104 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-certs\") pod \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " Aug 04 00:45:07.739136 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739135 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-socket\") pod \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " Aug 04 00:45:07.739358 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739160 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-credential-socket\") pod \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " Aug 04 00:45:07.739358 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739195 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-envoy\") pod \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " Aug 04 00:45:07.739358 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739229 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-token\") pod \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " Aug 04 00:45:07.739358 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739285 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-data\") pod \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " Aug 04 00:45:07.739358 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739339 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-podinfo\") pod \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " Aug 04 00:45:07.739615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739367 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvllf\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-kube-api-access-cvllf\") pod \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " Aug 04 00:45:07.739615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739399 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istiod-ca-cert\") pod \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\" (UID: \"1cdabc39-32fb-4a74-a6cd-d032d896e9b0\") " Aug 04 00:45:07.739615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739415 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cdabc39-32fb-4a74-a6cd-d032d896e9b0" (UID: "1cdabc39-32fb-4a74-a6cd-d032d896e9b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:45:07.739615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739479 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cdabc39-32fb-4a74-a6cd-d032d896e9b0" (UID: "1cdabc39-32fb-4a74-a6cd-d032d896e9b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:45:07.739615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739498 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cdabc39-32fb-4a74-a6cd-d032d896e9b0" (UID: "1cdabc39-32fb-4a74-a6cd-d032d896e9b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:45:07.739834 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739783 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:45:07.739834 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739804 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:45:07.739834 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739819 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:45:07.739834 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739782 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cdabc39-32fb-4a74-a6cd-d032d896e9b0" (UID: "1cdabc39-32fb-4a74-a6cd-d032d896e9b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:45:07.740013 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.739924 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cdabc39-32fb-4a74-a6cd-d032d896e9b0" (UID: "1cdabc39-32fb-4a74-a6cd-d032d896e9b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:45:07.741457 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.741424 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cdabc39-32fb-4a74-a6cd-d032d896e9b0" (UID: "1cdabc39-32fb-4a74-a6cd-d032d896e9b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:45:07.741593 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.741539 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-kube-api-access-cvllf" (OuterVolumeSpecName: "kube-api-access-cvllf") pod "1cdabc39-32fb-4a74-a6cd-d032d896e9b0" (UID: "1cdabc39-32fb-4a74-a6cd-d032d896e9b0"). InnerVolumeSpecName "kube-api-access-cvllf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:45:07.741818 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.741792 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cdabc39-32fb-4a74-a6cd-d032d896e9b0" (UID: "1cdabc39-32fb-4a74-a6cd-d032d896e9b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:45:07.741818 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.741806 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cdabc39-32fb-4a74-a6cd-d032d896e9b0" (UID: "1cdabc39-32fb-4a74-a6cd-d032d896e9b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:45:07.840501 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.840473 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:45:07.840501 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.840497 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:45:07.840501 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.840506 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:45:07.840700 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.840516 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:45:07.840700 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.840524 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvllf\" (UniqueName: \"kubernetes.io/projected/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-kube-api-access-cvllf\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:45:07.840700 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:07.840534 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cdabc39-32fb-4a74-a6cd-d032d896e9b0-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:45:08.496050 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.496013 2552 generic.go:356] "Generic (PLEG): container finished" podID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerID="aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110" exitCode=0 Aug 04 00:45:08.496223 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.496062 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" Aug 04 00:45:08.496223 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.496079 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" event={"ID":"1cdabc39-32fb-4a74-a6cd-d032d896e9b0","Type":"ContainerDied","Data":"aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110"} Aug 04 00:45:08.496223 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.496129 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p" event={"ID":"1cdabc39-32fb-4a74-a6cd-d032d896e9b0","Type":"ContainerDied","Data":"5787ba0323fe64f8209e6830bd4ac5a084db2694a3e30ceb21a109705417c9c7"} Aug 04 00:45:08.496223 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.496156 2552 scope.go:117] "RemoveContainer" containerID="aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110" Aug 04 00:45:08.505049 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.505032 2552 scope.go:117] "RemoveContainer" containerID="4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030" Aug 04 00:45:08.513375 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.513354 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p"] Aug 04 00:45:08.513441 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.513377 2552 scope.go:117] "RemoveContainer" containerID="aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110" Aug 04 00:45:08.513643 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:45:08.513625 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110\": container with ID starting with aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110 not found: ID does not exist" containerID="aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110" Aug 04 00:45:08.513694 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.513650 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110"} err="failed to get container status \"aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110\": rpc error: code = NotFound desc = could not find container \"aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110\": container with ID starting with aff0afc25b391e50317fd2ebcb38a9dc90c51b2b6d4018e13a09224906d3d110 not found: ID does not exist" Aug 04 00:45:08.513694 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.513667 2552 scope.go:117] "RemoveContainer" containerID="4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030" Aug 04 00:45:08.513944 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:45:08.513927 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030\": container with ID starting with 4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030 not found: ID does not exist" containerID="4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030" Aug 04 00:45:08.514002 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.513950 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030"} err="failed to get container status \"4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030\": rpc error: code = NotFound desc = could not find container \"4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030\": container with ID starting with 4f2a8a113e77bf0f2f5b259bceda1e91e6b506df41b1768f90ef1aa961394030 not found: ID does not exist" Aug 04 00:45:08.517191 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:08.517171 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-746fe6d0-openshift-default-5f5c8c56dd-6sf4p"] Aug 04 00:45:09.952875 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:45:09.952837 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" path="/var/lib/kubelet/pods/1cdabc39-32fb-4a74-a6cd-d032d896e9b0/volumes" Aug 04 00:46:07.350580 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.350505 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq"] Aug 04 00:46:07.350974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.350874 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" Aug 04 00:46:07.350974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.350886 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" Aug 04 00:46:07.350974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.350894 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" Aug 04 00:46:07.350974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.350900 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" Aug 04 00:46:07.350974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.350920 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" containerName="main" Aug 04 00:46:07.350974 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.350925 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" containerName="main" Aug 04 00:46:07.351175 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.350986 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" Aug 04 00:46:07.351175 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.350996 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="00151e37-78f5-4ef5-bbe6-3c3f5753ce5e" containerName="main" Aug 04 00:46:07.354235 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.354215 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.356843 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.356820 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4f7c8d-openshift-default-dockercfg-dvmlk\"" Aug 04 00:46:07.367208 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.367185 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq"] Aug 04 00:46:07.453487 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.453435 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-envoy\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.453711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.453501 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572a257e-f95a-45cc-9faa-6ac46724b8fd-istiod-ca-cert\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.453711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.453572 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-certs\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.453711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.453626 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-data\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.453711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.453682 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zsfg\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-kube-api-access-2zsfg\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.453962 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.453725 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-socket\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.453962 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.453748 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-token\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.453962 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.453796 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-credential-socket\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.453962 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.453883 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-podinfo\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554356 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554320 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zsfg\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-kube-api-access-2zsfg\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554517 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554368 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-socket\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554517 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554475 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-token\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554608 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554522 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-credential-socket\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554608 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554561 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-podinfo\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554614 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-envoy\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554651 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572a257e-f95a-45cc-9faa-6ac46724b8fd-istiod-ca-cert\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554820 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554732 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-certs\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554820 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554754 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-data\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.554965 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554893 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-socket\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.555023 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.554995 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-credential-socket\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.555197 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.555164 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-certs\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.555197 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.555174 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-data\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.555443 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.555379 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572a257e-f95a-45cc-9faa-6ac46724b8fd-istiod-ca-cert\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.556888 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.556848 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-envoy\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.557101 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.557083 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-podinfo\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.562362 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.562343 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-token\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.562745 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.562725 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zsfg\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-kube-api-access-2zsfg\") pod \"e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.664021 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.663989 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:07.806024 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.805990 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq"] Aug 04 00:46:07.812004 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.811804 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:07.812004 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.811980 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:07.812134 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:07.812074 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:08.786039 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:08.785998 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" event={"ID":"572a257e-f95a-45cc-9faa-6ac46724b8fd","Type":"ContainerStarted","Data":"42a72e916f85e68d33834409c0e514b21b781db8d9985ebaf15739f2aaa97274"} Aug 04 00:46:08.786431 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:08.786056 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" event={"ID":"572a257e-f95a-45cc-9faa-6ac46724b8fd","Type":"ContainerStarted","Data":"5f1f1f2ff88375bbd7d441fafc130993ff08cd60b1090a72e0db0937a088d491"} Aug 04 00:46:08.805905 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:08.805820 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" podStartSLOduration=1.805805521 podStartE2EDuration="1.805805521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:46:08.805525316 +0000 UTC m=+2607.468434817" watchObservedRunningTime="2026-08-04 00:46:08.805805521 +0000 UTC m=+2607.468714998" Aug 04 00:46:09.664720 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:09.664683 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:09.669555 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:09.669528 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:09.791395 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:09.791361 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:09.792349 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:09.792331 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:24.793059 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:24.793021 2552 patch_prober.go:28] interesting pod/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 00:46:24.793471 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:24.793093 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 00:46:24.862039 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:24.862005 2552 generic.go:356] "Generic (PLEG): container finished" podID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerID="42a72e916f85e68d33834409c0e514b21b781db8d9985ebaf15739f2aaa97274" exitCode=0 Aug 04 00:46:24.862200 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:24.862077 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" event={"ID":"572a257e-f95a-45cc-9faa-6ac46724b8fd","Type":"ContainerDied","Data":"42a72e916f85e68d33834409c0e514b21b781db8d9985ebaf15739f2aaa97274"} Aug 04 00:46:24.862480 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:24.862464 2552 scope.go:117] "RemoveContainer" containerID="42a72e916f85e68d33834409c0e514b21b781db8d9985ebaf15739f2aaa97274" Aug 04 00:46:24.863487 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:24.863456 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:24.863607 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:24.863528 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:24.863607 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:24.863563 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:25.869146 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:25.869106 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" event={"ID":"572a257e-f95a-45cc-9faa-6ac46724b8fd","Type":"ContainerStarted","Data":"ee14642161ed05014b3dc2d4d3aa7f1f7166b86f5282261cbe893b0c63967f62"} Aug 04 00:46:26.665027 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:26.664993 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:26.665291 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:26.665265 2552 patch_prober.go:28] interesting pod/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 00:46:26.665388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:26.665327 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 00:46:27.664602 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:27.664565 2552 patch_prober.go:28] interesting pod/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 00:46:27.665079 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:27.664626 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 00:46:28.665296 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:28.665262 2552 patch_prober.go:28] interesting pod/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 00:46:28.665699 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:28.665320 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 00:46:29.664970 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:29.664930 2552 patch_prober.go:28] interesting pod/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 00:46:29.665145 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:29.664991 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 00:46:30.664916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:30.664880 2552 patch_prober.go:28] interesting pod/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 00:46:30.665297 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:30.664944 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 00:46:31.668813 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:31.668783 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:31.669234 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:31.669127 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:31.669716 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:31.669702 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:46:36.319834 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.319796 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz"] Aug 04 00:46:36.321005 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.320978 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdabc39-32fb-4a74-a6cd-d032d896e9b0" containerName="istio-proxy" Aug 04 00:46:36.323281 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.323259 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.326040 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.326014 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-743d5d-openshift-default-dockercfg-9dn47\"" Aug 04 00:46:36.335587 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.335565 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz"] Aug 04 00:46:36.406928 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.406896 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-token\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.407099 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.406932 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-podinfo\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.407099 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.406960 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istiod-ca-cert\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.407099 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.407001 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-envoy\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.407099 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.407034 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-socket\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.407263 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.407106 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-certs\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.407263 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.407134 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-data\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.407263 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.407169 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9mt\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-kube-api-access-6w9mt\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.407263 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.407193 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-credential-socket\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508233 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508193 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-token\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508493 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508240 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-podinfo\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508493 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508277 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istiod-ca-cert\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508493 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508304 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-envoy\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508493 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508344 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-socket\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508493 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508398 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-certs\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508493 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508425 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-data\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508493 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508473 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9mt\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-kube-api-access-6w9mt\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508833 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508506 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-credential-socket\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508833 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508801 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-data\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508993 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508851 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-socket\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.508993 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.508923 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-certs\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.509091 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.509017 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-credential-socket\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.509128 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.509110 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istiod-ca-cert\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.511207 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.511184 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-envoy\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.511405 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.511388 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-podinfo\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.516410 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.516388 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-token\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.516678 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.516656 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9mt\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-kube-api-access-6w9mt\") pod \"e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.635480 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.635448 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:36.767847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.767822 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz"] Aug 04 00:46:36.770068 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:46:36.770018 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df2ac31_792a_47ab_b352_942ad9cdb3c4.slice/crio-c627e73f2c34a8eee77a10fb529212428746f46a4000fb0b93ddb98b62c672ae WatchSource:0}: Error finding container c627e73f2c34a8eee77a10fb529212428746f46a4000fb0b93ddb98b62c672ae: Status 404 returned error can't find the container with id c627e73f2c34a8eee77a10fb529212428746f46a4000fb0b93ddb98b62c672ae Aug 04 00:46:36.773013 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.772974 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:36.773122 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.773061 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:36.773122 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.773104 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:36.946184 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.946094 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" event={"ID":"6df2ac31-792a-47ab-b352-942ad9cdb3c4","Type":"ContainerStarted","Data":"a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108"} Aug 04 00:46:36.946184 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.946170 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" event={"ID":"6df2ac31-792a-47ab-b352-942ad9cdb3c4","Type":"ContainerStarted","Data":"c627e73f2c34a8eee77a10fb529212428746f46a4000fb0b93ddb98b62c672ae"} Aug 04 00:46:36.965824 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:36.965764 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" podStartSLOduration=0.965749534 podStartE2EDuration="965.749534ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:46:36.964137734 +0000 UTC m=+2635.627047232" watchObservedRunningTime="2026-08-04 00:46:36.965749534 +0000 UTC m=+2635.628659012" Aug 04 00:46:37.635925 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:37.635890 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:37.640793 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:37.640767 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:37.953261 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:37.953187 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:37.953261 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:37.953251 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:40.591995 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.591961 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5"] Aug 04 00:46:40.594455 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.594440 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:40.596952 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.596930 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-743d5d\"" Aug 04 00:46:40.597057 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.596964 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-743d5d-dockercfg-zpqrs\"" Aug 04 00:46:40.604674 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.604653 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5"] Aug 04 00:46:40.647076 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.647045 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999tn\" (UniqueName: \"kubernetes.io/projected/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-kube-api-access-999tn\") pod \"payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5\" (UID: \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:40.647257 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.647102 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5\" (UID: \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:40.748595 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.748557 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999tn\" (UniqueName: \"kubernetes.io/projected/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-kube-api-access-999tn\") pod \"payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5\" (UID: \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:40.748773 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.748621 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5\" (UID: \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:40.749297 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.749277 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5\" (UID: \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:40.757734 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.757706 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-999tn\" (UniqueName: \"kubernetes.io/projected/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-kube-api-access-999tn\") pod \"payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5\" (UID: \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:40.906700 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:40.906666 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:41.040236 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:41.040211 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5"] Aug 04 00:46:41.042615 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:46:41.042586 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f8a672_ee7e_4ed9_9a52_6c17f4c9e6e9.slice/crio-278e3fd76c3639eda7b3be462a92d7ba4c2a2ec02d9f9fdf183bbb86d5226b41 WatchSource:0}: Error finding container 278e3fd76c3639eda7b3be462a92d7ba4c2a2ec02d9f9fdf183bbb86d5226b41: Status 404 returned error can't find the container with id 278e3fd76c3639eda7b3be462a92d7ba4c2a2ec02d9f9fdf183bbb86d5226b41 Aug 04 00:46:41.976093 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:41.976051 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" event={"ID":"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9","Type":"ContainerStarted","Data":"6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34"} Aug 04 00:46:41.976494 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:41.976117 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" event={"ID":"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9","Type":"ContainerStarted","Data":"278e3fd76c3639eda7b3be462a92d7ba4c2a2ec02d9f9fdf183bbb86d5226b41"} Aug 04 00:46:41.976494 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:41.976186 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:41.994350 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:41.994297 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" podStartSLOduration=1.994280436 podStartE2EDuration="1.994280436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:46:41.992074219 +0000 UTC m=+2640.654983689" watchObservedRunningTime="2026-08-04 00:46:41.994280436 +0000 UTC m=+2640.657189914" Aug 04 00:46:42.181148 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:42.181116 2552 scope.go:117] "RemoveContainer" containerID="7addc2c97104a9de1053d136c61f234fc4a1d68d64efc07be192079437cbd56a" Aug 04 00:46:52.033365 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:52.033269 2552 generic.go:356] "Generic (PLEG): container finished" podID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerID="a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108" exitCode=0 Aug 04 00:46:52.033365 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:52.033340 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" event={"ID":"6df2ac31-792a-47ab-b352-942ad9cdb3c4","Type":"ContainerDied","Data":"a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108"} Aug 04 00:46:52.033789 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:52.033716 2552 scope.go:117] "RemoveContainer" containerID="a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108" Aug 04 00:46:52.034795 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:52.034770 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:52.034896 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:52.034850 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:52.034944 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:52.034923 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:46:52.952034 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:52.952003 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:52.984388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:52.984347 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:46:53.040696 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:53.040664 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" event={"ID":"6df2ac31-792a-47ab-b352-942ad9cdb3c4","Type":"ContainerStarted","Data":"97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948"} Aug 04 00:46:53.635754 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:53.635714 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:53.636712 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:53.636686 2552 patch_prober.go:28] interesting pod/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 00:46:53.636823 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:53.636744 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 00:46:54.636462 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:54.636423 2552 patch_prober.go:28] interesting pod/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 00:46:54.636892 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:54.636483 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 00:46:55.636553 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:55.636515 2552 patch_prober.go:28] interesting pod/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 00:46:55.636972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:55.636591 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 00:46:56.636187 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:56.636149 2552 patch_prober.go:28] interesting pod/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 00:46:56.636399 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:56.636214 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 00:46:57.635998 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:57.635964 2552 patch_prober.go:28] interesting pod/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 00:46:57.636417 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:57.636021 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 00:46:58.640567 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:58.640536 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:58.641064 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:58.640792 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:46:58.641560 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:46:58.641545 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:47:07.438586 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.438550 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg"] Aug 04 00:47:07.442727 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.442706 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.445351 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.445333 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4f7c8d\"/\"openshift-service-ca.crt\"" Aug 04 00:47:07.446359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.446336 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4f7c8d\"/\"default-dockercfg-wfdm6\"" Aug 04 00:47:07.446467 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.446333 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4f7c8d\"/\"test-model-4f7c8d-kserve-self-signed-certs\"" Aug 04 00:47:07.446467 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.446371 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4f7c8d\"/\"kube-root-ca.crt\"" Aug 04 00:47:07.451796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.451656 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg"] Aug 04 00:47:07.602580 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.602545 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-model-cache\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.602580 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.602580 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-dshm\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.602783 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.602605 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-home\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.602783 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.602716 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94181cde-8520-40dd-8604-cad3e41297ae-tls-certs\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.602783 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.602753 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-tmp-dir\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.602783 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.602772 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvqw6\" (UniqueName: \"kubernetes.io/projected/94181cde-8520-40dd-8604-cad3e41297ae-kube-api-access-wvqw6\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.703809 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.703730 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-model-cache\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.703809 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.703762 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-dshm\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.703809 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.703790 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-home\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.704113 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.703831 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94181cde-8520-40dd-8604-cad3e41297ae-tls-certs\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.704113 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.703853 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-tmp-dir\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.704113 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.703903 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvqw6\" (UniqueName: \"kubernetes.io/projected/94181cde-8520-40dd-8604-cad3e41297ae-kube-api-access-wvqw6\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.704281 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.704249 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-home\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.704339 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.704303 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-tmp-dir\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.704339 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.704327 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-model-cache\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.706066 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.706043 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-dshm\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.706318 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.706301 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94181cde-8520-40dd-8604-cad3e41297ae-tls-certs\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.711695 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.711674 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvqw6\" (UniqueName: \"kubernetes.io/projected/94181cde-8520-40dd-8604-cad3e41297ae-kube-api-access-wvqw6\") pod \"test-model-4f7c8d-kserve-766bc4db85-97llg\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.755314 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.755289 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:07.886104 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:07.886069 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg"] Aug 04 00:47:07.889022 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:47:07.888993 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94181cde_8520_40dd_8604_cad3e41297ae.slice/crio-e6ed6ea08bca2801dc3a71d814a2160453a2190c3c90be3bd3c0b87b88897cd4 WatchSource:0}: Error finding container e6ed6ea08bca2801dc3a71d814a2160453a2190c3c90be3bd3c0b87b88897cd4: Status 404 returned error can't find the container with id e6ed6ea08bca2801dc3a71d814a2160453a2190c3c90be3bd3c0b87b88897cd4 Aug 04 00:47:08.116240 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:08.116209 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" event={"ID":"94181cde-8520-40dd-8604-cad3e41297ae","Type":"ContainerStarted","Data":"5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0"} Aug 04 00:47:08.116413 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:08.116249 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" event={"ID":"94181cde-8520-40dd-8604-cad3e41297ae","Type":"ContainerStarted","Data":"e6ed6ea08bca2801dc3a71d814a2160453a2190c3c90be3bd3c0b87b88897cd4"} Aug 04 00:47:08.132270 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:08.132222 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" podStartSLOduration=1.132203815 podStartE2EDuration="1.132203815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:47:08.131745281 +0000 UTC m=+2666.794654761" watchObservedRunningTime="2026-08-04 00:47:08.132203815 +0000 UTC m=+2666.795113294" Aug 04 00:47:17.756292 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:17.756252 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:17.756292 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:17.756300 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:17.768905 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:17.768878 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:47:18.173874 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:47:18.173829 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:48:13.012603 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:13.012564 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg"] Aug 04 00:48:28.035889 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.035803 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" podUID="94181cde-8520-40dd-8604-cad3e41297ae" containerName="main" containerID="cri-o://5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0" gracePeriod=45 Aug 04 00:48:28.163527 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.163467 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" podUID="94181cde-8520-40dd-8604-cad3e41297ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": dial tcp 10.133.0.62:8000: connect: connection refused" Aug 04 00:48:28.287901 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.287810 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:48:28.358933 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.358901 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94181cde-8520-40dd-8604-cad3e41297ae-tls-certs\") pod \"94181cde-8520-40dd-8604-cad3e41297ae\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " Aug 04 00:48:28.359140 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.358953 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-model-cache\") pod \"94181cde-8520-40dd-8604-cad3e41297ae\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " Aug 04 00:48:28.359140 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.358985 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-dshm\") pod \"94181cde-8520-40dd-8604-cad3e41297ae\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " Aug 04 00:48:28.359140 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.359002 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvqw6\" (UniqueName: \"kubernetes.io/projected/94181cde-8520-40dd-8604-cad3e41297ae-kube-api-access-wvqw6\") pod \"94181cde-8520-40dd-8604-cad3e41297ae\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " Aug 04 00:48:28.359140 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.359048 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-home\") pod \"94181cde-8520-40dd-8604-cad3e41297ae\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " Aug 04 00:48:28.359140 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.359098 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-tmp-dir\") pod \"94181cde-8520-40dd-8604-cad3e41297ae\" (UID: \"94181cde-8520-40dd-8604-cad3e41297ae\") " Aug 04 00:48:28.359404 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.359241 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "94181cde-8520-40dd-8604-cad3e41297ae" (UID: "94181cde-8520-40dd-8604-cad3e41297ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:28.359478 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.359427 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-home" (OuterVolumeSpecName: "home") pod "94181cde-8520-40dd-8604-cad3e41297ae" (UID: "94181cde-8520-40dd-8604-cad3e41297ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:28.359478 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.359459 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:28.359588 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.359439 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94181cde-8520-40dd-8604-cad3e41297ae" (UID: "94181cde-8520-40dd-8604-cad3e41297ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:28.361285 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.361262 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-dshm" (OuterVolumeSpecName: "dshm") pod "94181cde-8520-40dd-8604-cad3e41297ae" (UID: "94181cde-8520-40dd-8604-cad3e41297ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:28.361408 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.361361 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94181cde-8520-40dd-8604-cad3e41297ae-kube-api-access-wvqw6" (OuterVolumeSpecName: "kube-api-access-wvqw6") pod "94181cde-8520-40dd-8604-cad3e41297ae" (UID: "94181cde-8520-40dd-8604-cad3e41297ae"). InnerVolumeSpecName "kube-api-access-wvqw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:28.361448 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.361433 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94181cde-8520-40dd-8604-cad3e41297ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94181cde-8520-40dd-8604-cad3e41297ae" (UID: "94181cde-8520-40dd-8604-cad3e41297ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:48:28.460124 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.460092 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94181cde-8520-40dd-8604-cad3e41297ae-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:28.460124 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.460117 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:28.460124 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.460126 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvqw6\" (UniqueName: \"kubernetes.io/projected/94181cde-8520-40dd-8604-cad3e41297ae-kube-api-access-wvqw6\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:28.460124 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.460135 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:28.460389 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.460144 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94181cde-8520-40dd-8604-cad3e41297ae-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:28.527128 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.527097 2552 generic.go:356] "Generic (PLEG): container finished" podID="94181cde-8520-40dd-8604-cad3e41297ae" containerID="5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0" exitCode=0 Aug 04 00:48:28.527306 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.527154 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" Aug 04 00:48:28.527306 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.527186 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" event={"ID":"94181cde-8520-40dd-8604-cad3e41297ae","Type":"ContainerDied","Data":"5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0"} Aug 04 00:48:28.527306 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.527251 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg" event={"ID":"94181cde-8520-40dd-8604-cad3e41297ae","Type":"ContainerDied","Data":"e6ed6ea08bca2801dc3a71d814a2160453a2190c3c90be3bd3c0b87b88897cd4"} Aug 04 00:48:28.527306 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.527273 2552 scope.go:117] "RemoveContainer" containerID="5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0" Aug 04 00:48:28.536186 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.536171 2552 scope.go:117] "RemoveContainer" containerID="5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0" Aug 04 00:48:28.536441 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:48:28.536423 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0\": container with ID starting with 5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0 not found: ID does not exist" containerID="5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0" Aug 04 00:48:28.536488 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.536450 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0"} err="failed to get container status \"5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0\": rpc error: code = NotFound desc = could not find container \"5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0\": container with ID starting with 5c92fbc6e6d94ad2122d8f02d9b1492a7e55ec5497414bf2fbada70af1f93fe0 not found: ID does not exist" Aug 04 00:48:28.549174 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.549115 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg"] Aug 04 00:48:28.553029 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:28.553004 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4f7c8d/test-model-4f7c8d-kserve-766bc4db85-97llg"] Aug 04 00:48:29.953852 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:29.953819 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94181cde-8520-40dd-8604-cad3e41297ae" path="/var/lib/kubelet/pods/94181cde-8520-40dd-8604-cad3e41297ae/volumes" Aug 04 00:48:30.405812 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:30.405766 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq"] Aug 04 00:48:30.406041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:30.406012 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" containerID="cri-o://ee14642161ed05014b3dc2d4d3aa7f1f7166b86f5282261cbe893b0c63967f62" gracePeriod=30 Aug 04 00:48:31.671683 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:31.671648 2552 patch_prober.go:28] interesting pod/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 00:48:31.672143 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:31.671710 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:48:35.566713 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.566678 2552 generic.go:356] "Generic (PLEG): container finished" podID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerID="ee14642161ed05014b3dc2d4d3aa7f1f7166b86f5282261cbe893b0c63967f62" exitCode=0 Aug 04 00:48:35.567137 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.566713 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" event={"ID":"572a257e-f95a-45cc-9faa-6ac46724b8fd","Type":"ContainerDied","Data":"ee14642161ed05014b3dc2d4d3aa7f1f7166b86f5282261cbe893b0c63967f62"} Aug 04 00:48:35.567137 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.566792 2552 scope.go:117] "RemoveContainer" containerID="42a72e916f85e68d33834409c0e514b21b781db8d9985ebaf15739f2aaa97274" Aug 04 00:48:35.664275 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.664247 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:48:35.717645 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.717582 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zsfg\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-kube-api-access-2zsfg\") pod \"572a257e-f95a-45cc-9faa-6ac46724b8fd\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " Aug 04 00:48:35.717645 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.717624 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-envoy\") pod \"572a257e-f95a-45cc-9faa-6ac46724b8fd\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " Aug 04 00:48:35.717841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.717663 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-credential-socket\") pod \"572a257e-f95a-45cc-9faa-6ac46724b8fd\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " Aug 04 00:48:35.717841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.717688 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-data\") pod \"572a257e-f95a-45cc-9faa-6ac46724b8fd\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " Aug 04 00:48:35.717841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.717715 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-token\") pod \"572a257e-f95a-45cc-9faa-6ac46724b8fd\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " Aug 04 00:48:35.717841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.717761 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-podinfo\") pod \"572a257e-f95a-45cc-9faa-6ac46724b8fd\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " Aug 04 00:48:35.717841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.717788 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572a257e-f95a-45cc-9faa-6ac46724b8fd-istiod-ca-cert\") pod \"572a257e-f95a-45cc-9faa-6ac46724b8fd\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " Aug 04 00:48:35.718135 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.717882 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-certs\") pod \"572a257e-f95a-45cc-9faa-6ac46724b8fd\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " Aug 04 00:48:35.718135 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.717915 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-socket\") pod \"572a257e-f95a-45cc-9faa-6ac46724b8fd\" (UID: \"572a257e-f95a-45cc-9faa-6ac46724b8fd\") " Aug 04 00:48:35.718135 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.718021 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "572a257e-f95a-45cc-9faa-6ac46724b8fd" (UID: "572a257e-f95a-45cc-9faa-6ac46724b8fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:35.718297 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.718200 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "572a257e-f95a-45cc-9faa-6ac46724b8fd" (UID: "572a257e-f95a-45cc-9faa-6ac46724b8fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:35.718359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.718310 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:35.718359 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.718327 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:35.718466 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.718387 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "572a257e-f95a-45cc-9faa-6ac46724b8fd" (UID: "572a257e-f95a-45cc-9faa-6ac46724b8fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:35.718466 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.718434 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "572a257e-f95a-45cc-9faa-6ac46724b8fd" (UID: "572a257e-f95a-45cc-9faa-6ac46724b8fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:35.718569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.718304 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572a257e-f95a-45cc-9faa-6ac46724b8fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "572a257e-f95a-45cc-9faa-6ac46724b8fd" (UID: "572a257e-f95a-45cc-9faa-6ac46724b8fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:35.719910 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.719886 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "572a257e-f95a-45cc-9faa-6ac46724b8fd" (UID: "572a257e-f95a-45cc-9faa-6ac46724b8fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:48:35.720007 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.719917 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "572a257e-f95a-45cc-9faa-6ac46724b8fd" (UID: "572a257e-f95a-45cc-9faa-6ac46724b8fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:35.720168 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.720150 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "572a257e-f95a-45cc-9faa-6ac46724b8fd" (UID: "572a257e-f95a-45cc-9faa-6ac46724b8fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:35.720230 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.720167 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-kube-api-access-2zsfg" (OuterVolumeSpecName: "kube-api-access-2zsfg") pod "572a257e-f95a-45cc-9faa-6ac46724b8fd" (UID: "572a257e-f95a-45cc-9faa-6ac46724b8fd"). InnerVolumeSpecName "kube-api-access-2zsfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:35.818752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.818724 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zsfg\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-kube-api-access-2zsfg\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:35.818752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.818748 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:35.818752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.818758 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:35.818978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.818767 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:35.818978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.818775 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572a257e-f95a-45cc-9faa-6ac46724b8fd-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:35.818978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.818783 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572a257e-f95a-45cc-9faa-6ac46724b8fd-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:35.818978 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:35.818792 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572a257e-f95a-45cc-9faa-6ac46724b8fd-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:36.573613 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:36.573580 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" Aug 04 00:48:36.573613 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:36.573594 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq" event={"ID":"572a257e-f95a-45cc-9faa-6ac46724b8fd","Type":"ContainerDied","Data":"5f1f1f2ff88375bbd7d441fafc130993ff08cd60b1090a72e0db0937a088d491"} Aug 04 00:48:36.574108 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:36.573657 2552 scope.go:117] "RemoveContainer" containerID="ee14642161ed05014b3dc2d4d3aa7f1f7166b86f5282261cbe893b0c63967f62" Aug 04 00:48:36.592326 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:36.592298 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq"] Aug 04 00:48:36.595840 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:36.595815 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4f7c8d-openshift-default-d55b8ddb6-xknxq"] Aug 04 00:48:37.958747 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:37.958708 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" path="/var/lib/kubelet/pods/572a257e-f95a-45cc-9faa-6ac46724b8fd/volumes" Aug 04 00:48:41.234109 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.234075 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5"] Aug 04 00:48:41.234520 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.234325 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" podUID="66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9" containerName="payload-processing" containerID="cri-o://6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34" gracePeriod=30 Aug 04 00:48:41.495438 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.495384 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:48:41.570900 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.570850 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-999tn\" (UniqueName: \"kubernetes.io/projected/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-kube-api-access-999tn\") pod \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\" (UID: \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\") " Aug 04 00:48:41.571085 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.570935 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-plugins-config-volume\") pod \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\" (UID: \"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9\") " Aug 04 00:48:41.571421 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.571389 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9" (UID: "66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:41.573448 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.573407 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-kube-api-access-999tn" (OuterVolumeSpecName: "kube-api-access-999tn") pod "66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9" (UID: "66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9"). InnerVolumeSpecName "kube-api-access-999tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:41.600507 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.600481 2552 generic.go:356] "Generic (PLEG): container finished" podID="66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9" containerID="6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34" exitCode=0 Aug 04 00:48:41.600615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.600521 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" Aug 04 00:48:41.600615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.600574 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" event={"ID":"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9","Type":"ContainerDied","Data":"6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34"} Aug 04 00:48:41.600748 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.600634 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5" event={"ID":"66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9","Type":"ContainerDied","Data":"278e3fd76c3639eda7b3be462a92d7ba4c2a2ec02d9f9fdf183bbb86d5226b41"} Aug 04 00:48:41.600748 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.600667 2552 scope.go:117] "RemoveContainer" containerID="6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34" Aug 04 00:48:41.610555 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.610538 2552 scope.go:117] "RemoveContainer" containerID="6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34" Aug 04 00:48:41.610807 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:48:41.610787 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34\": container with ID starting with 6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34 not found: ID does not exist" containerID="6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34" Aug 04 00:48:41.610899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.610821 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34"} err="failed to get container status \"6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34\": rpc error: code = NotFound desc = could not find container \"6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34\": container with ID starting with 6fc6c7ad7fff8e7b41758c6726f267840276fb715032e037b47829bcf3d3ec34 not found: ID does not exist" Aug 04 00:48:41.621020 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.620998 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5"] Aug 04 00:48:41.624598 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.624577 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-743d5d-56b74759c8-frtm5"] Aug 04 00:48:41.672770 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.672734 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-999tn\" (UniqueName: \"kubernetes.io/projected/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-kube-api-access-999tn\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:41.672770 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.672767 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:41.959425 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:41.959383 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9" path="/var/lib/kubelet/pods/66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9/volumes" Aug 04 00:48:47.914283 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:47.914250 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz"] Aug 04 00:48:47.914773 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:47.914498 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" containerID="cri-o://97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948" gracePeriod=30 Aug 04 00:48:48.828513 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.828479 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf"] Aug 04 00:48:48.828879 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.828851 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" Aug 04 00:48:48.828879 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.828877 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" Aug 04 00:48:48.828972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.828889 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9" containerName="payload-processing" Aug 04 00:48:48.828972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.828896 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9" containerName="payload-processing" Aug 04 00:48:48.828972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.828906 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94181cde-8520-40dd-8604-cad3e41297ae" containerName="main" Aug 04 00:48:48.828972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.828915 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="94181cde-8520-40dd-8604-cad3e41297ae" containerName="main" Aug 04 00:48:48.829097 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.828980 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f8a672-ee7e-4ed9-9a52-6c17f4c9e6e9" containerName="payload-processing" Aug 04 00:48:48.829097 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.828993 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" Aug 04 00:48:48.829097 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.829001 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="94181cde-8520-40dd-8604-cad3e41297ae" containerName="main" Aug 04 00:48:48.833599 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.833581 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:48.836613 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.836589 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a12ead-openshift-default-dockercfg-7nhqv\"" Aug 04 00:48:48.843153 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.843131 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf"] Aug 04 00:48:48.940229 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.940185 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-data\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:48.940229 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.940230 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttr4\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-kube-api-access-tttr4\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:48.940766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.940285 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-credential-socket\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:48.940766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.940318 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec0d949-090a-496c-8965-55c5df56b9f4-istio-podinfo\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:48.940766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.940345 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-istio-token\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:48.940766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.940373 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec0d949-090a-496c-8965-55c5df56b9f4-istiod-ca-cert\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:48.940766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.940451 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-socket\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:48.940766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.940497 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-envoy\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:48.940766 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:48.940599 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-certs\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041438 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041401 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-envoy\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041641 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041500 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-certs\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041641 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041564 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-data\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041641 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041584 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttr4\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-kube-api-access-tttr4\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041641 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041610 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-credential-socket\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041641 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041635 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec0d949-090a-496c-8965-55c5df56b9f4-istio-podinfo\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041966 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041651 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-istio-token\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041966 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041668 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec0d949-090a-496c-8965-55c5df56b9f4-istiod-ca-cert\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041966 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041724 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-socket\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.041966 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041917 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-certs\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.042181 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.041963 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-data\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.042240 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.042225 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-socket\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.042297 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.042252 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-credential-socket\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.042473 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.042454 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec0d949-090a-496c-8965-55c5df56b9f4-istiod-ca-cert\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.043660 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.043639 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-envoy\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.043961 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.043942 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec0d949-090a-496c-8965-55c5df56b9f4-istio-podinfo\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.049627 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.049605 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-istio-token\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.049756 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.049683 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttr4\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-kube-api-access-tttr4\") pod \"e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.145484 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.145451 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:49.286800 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.286775 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf"] Aug 04 00:48:49.288803 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:48:49.288767 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec0d949_090a_496c_8965_55c5df56b9f4.slice/crio-2b4d4517d94bfd6d01e6ef7be34d10a1fd58fb3f92f31ab8a4cdbe7b766c3354 WatchSource:0}: Error finding container 2b4d4517d94bfd6d01e6ef7be34d10a1fd58fb3f92f31ab8a4cdbe7b766c3354: Status 404 returned error can't find the container with id 2b4d4517d94bfd6d01e6ef7be34d10a1fd58fb3f92f31ab8a4cdbe7b766c3354 Aug 04 00:48:49.291330 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.291297 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:48:49.291476 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.291379 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:48:49.291476 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.291413 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:48:49.641434 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.641398 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" event={"ID":"aec0d949-090a-496c-8965-55c5df56b9f4","Type":"ContainerStarted","Data":"49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206"} Aug 04 00:48:49.641607 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.641464 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" event={"ID":"aec0d949-090a-496c-8965-55c5df56b9f4","Type":"ContainerStarted","Data":"2b4d4517d94bfd6d01e6ef7be34d10a1fd58fb3f92f31ab8a4cdbe7b766c3354"} Aug 04 00:48:49.660969 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:49.660926 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" podStartSLOduration=1.660911879 podStartE2EDuration="1.660911879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:48:49.660293784 +0000 UTC m=+2768.323203284" watchObservedRunningTime="2026-08-04 00:48:49.660911879 +0000 UTC m=+2768.323821356" Aug 04 00:48:50.146253 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:50.146218 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:50.151733 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:50.151705 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:50.646773 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:50.646745 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:50.647710 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:50.647687 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:48:53.160199 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.160172 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:48:53.277503 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277414 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-data\") pod \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " Aug 04 00:48:53.277503 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277462 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w9mt\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-kube-api-access-6w9mt\") pod \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " Aug 04 00:48:53.277785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277516 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-podinfo\") pod \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " Aug 04 00:48:53.277785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277566 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-certs\") pod \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " Aug 04 00:48:53.277785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277621 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-credential-socket\") pod \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " Aug 04 00:48:53.277785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277671 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istiod-ca-cert\") pod \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " Aug 04 00:48:53.277785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277721 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-socket\") pod \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " Aug 04 00:48:53.277785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277742 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "6df2ac31-792a-47ab-b352-942ad9cdb3c4" (UID: "6df2ac31-792a-47ab-b352-942ad9cdb3c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:53.277785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277756 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-token\") pod \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " Aug 04 00:48:53.278189 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277830 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-envoy\") pod \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\" (UID: \"6df2ac31-792a-47ab-b352-942ad9cdb3c4\") " Aug 04 00:48:53.278189 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.277843 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6df2ac31-792a-47ab-b352-942ad9cdb3c4" (UID: "6df2ac31-792a-47ab-b352-942ad9cdb3c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:53.278189 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.278039 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6df2ac31-792a-47ab-b352-942ad9cdb3c4" (UID: "6df2ac31-792a-47ab-b352-942ad9cdb3c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:53.278189 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.278125 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6df2ac31-792a-47ab-b352-942ad9cdb3c4" (UID: "6df2ac31-792a-47ab-b352-942ad9cdb3c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:53.278398 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.278186 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6df2ac31-792a-47ab-b352-942ad9cdb3c4" (UID: "6df2ac31-792a-47ab-b352-942ad9cdb3c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:53.278398 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.278215 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:53.278398 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.278234 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:53.278398 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.278251 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:53.278398 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.278264 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:53.280158 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.280124 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "6df2ac31-792a-47ab-b352-942ad9cdb3c4" (UID: "6df2ac31-792a-47ab-b352-942ad9cdb3c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:53.280158 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.280126 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-kube-api-access-6w9mt" (OuterVolumeSpecName: "kube-api-access-6w9mt") pod "6df2ac31-792a-47ab-b352-942ad9cdb3c4" (UID: "6df2ac31-792a-47ab-b352-942ad9cdb3c4"). InnerVolumeSpecName "kube-api-access-6w9mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:53.280303 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.280128 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6df2ac31-792a-47ab-b352-942ad9cdb3c4" (UID: "6df2ac31-792a-47ab-b352-942ad9cdb3c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:48:53.280303 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.280220 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6df2ac31-792a-47ab-b352-942ad9cdb3c4" (UID: "6df2ac31-792a-47ab-b352-942ad9cdb3c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:53.379460 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.379429 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:53.379460 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.379457 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:53.379460 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.379466 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:53.379684 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.379474 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df2ac31-792a-47ab-b352-942ad9cdb3c4-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:53.379684 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.379482 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w9mt\" (UniqueName: \"kubernetes.io/projected/6df2ac31-792a-47ab-b352-942ad9cdb3c4-kube-api-access-6w9mt\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:48:53.661406 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.661372 2552 generic.go:356] "Generic (PLEG): container finished" podID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerID="97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948" exitCode=0 Aug 04 00:48:53.661571 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.661422 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" Aug 04 00:48:53.661571 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.661453 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" event={"ID":"6df2ac31-792a-47ab-b352-942ad9cdb3c4","Type":"ContainerDied","Data":"97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948"} Aug 04 00:48:53.661571 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.661503 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz" event={"ID":"6df2ac31-792a-47ab-b352-942ad9cdb3c4","Type":"ContainerDied","Data":"c627e73f2c34a8eee77a10fb529212428746f46a4000fb0b93ddb98b62c672ae"} Aug 04 00:48:53.661571 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.661524 2552 scope.go:117] "RemoveContainer" containerID="97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948" Aug 04 00:48:53.670737 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.670716 2552 scope.go:117] "RemoveContainer" containerID="a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108" Aug 04 00:48:53.678627 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.678607 2552 scope.go:117] "RemoveContainer" containerID="97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948" Aug 04 00:48:53.678893 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:48:53.678866 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948\": container with ID starting with 97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948 not found: ID does not exist" containerID="97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948" Aug 04 00:48:53.678980 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.678899 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948"} err="failed to get container status \"97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948\": rpc error: code = NotFound desc = could not find container \"97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948\": container with ID starting with 97d035a0a635f0294ab7c3bdf2001e239be07f56bfb07c0daae7fce01ded6948 not found: ID does not exist" Aug 04 00:48:53.678980 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.678917 2552 scope.go:117] "RemoveContainer" containerID="a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108" Aug 04 00:48:53.679161 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:48:53.679147 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108\": container with ID starting with a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108 not found: ID does not exist" containerID="a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108" Aug 04 00:48:53.679203 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.679164 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108"} err="failed to get container status \"a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108\": rpc error: code = NotFound desc = could not find container \"a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108\": container with ID starting with a27704238edd995c6c0ab624c230682d92ba95f3ea235c2745fe04ad4de8a108 not found: ID does not exist" Aug 04 00:48:53.682746 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.682723 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz"] Aug 04 00:48:53.687247 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.687221 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-743d5d-openshift-default-f4b67c98b-c78fz"] Aug 04 00:48:53.953318 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:48:53.953247 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" path="/var/lib/kubelet/pods/6df2ac31-792a-47ab-b352-942ad9cdb3c4/volumes" Aug 04 00:49:04.724227 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:04.724190 2552 generic.go:356] "Generic (PLEG): container finished" podID="aec0d949-090a-496c-8965-55c5df56b9f4" containerID="49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206" exitCode=0 Aug 04 00:49:04.724666 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:04.724263 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" event={"ID":"aec0d949-090a-496c-8965-55c5df56b9f4","Type":"ContainerDied","Data":"49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206"} Aug 04 00:49:04.724720 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:04.724702 2552 scope.go:117] "RemoveContainer" containerID="49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206" Aug 04 00:49:04.725694 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:04.725665 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:49:04.725793 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:04.725743 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:49:04.725917 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:04.725790 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:49:05.646916 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:05.646841 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:49:05.731189 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:05.731144 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" event={"ID":"aec0d949-090a-496c-8965-55c5df56b9f4","Type":"ContainerStarted","Data":"3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4"} Aug 04 00:49:06.146169 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:06.146137 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:49:06.147421 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:06.147395 2552 patch_prober.go:28] interesting pod/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 00:49:06.147542 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:06.147443 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 00:49:07.146394 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:07.146360 2552 patch_prober.go:28] interesting pod/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 00:49:07.146785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:07.146415 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 00:49:08.145976 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:08.145942 2552 patch_prober.go:28] interesting pod/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 00:49:08.146156 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:08.145996 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 00:49:09.146256 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:09.146221 2552 patch_prober.go:28] interesting pod/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 00:49:09.146754 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:09.146275 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 00:49:10.146244 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:10.146207 2552 patch_prober.go:28] interesting pod/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 00:49:10.146422 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:10.146274 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 00:49:11.152207 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:11.152180 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:49:11.152647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:11.152626 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:49:11.153569 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:11.153547 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:49:17.260778 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.260744 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn"] Aug 04 00:49:17.261254 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.261231 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" Aug 04 00:49:17.261254 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.261254 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" Aug 04 00:49:17.261499 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.261281 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" Aug 04 00:49:17.261499 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.261291 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" Aug 04 00:49:17.261499 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.261302 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" Aug 04 00:49:17.261499 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.261311 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" Aug 04 00:49:17.261499 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.261458 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="572a257e-f95a-45cc-9faa-6ac46724b8fd" containerName="istio-proxy" Aug 04 00:49:17.261499 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.261477 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" Aug 04 00:49:17.263593 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.263576 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.266000 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.265979 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5782b0-openshift-default-dockercfg-7rfcv\"" Aug 04 00:49:17.276644 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.276619 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn"] Aug 04 00:49:17.299419 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.299391 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-socket\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.299572 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.299545 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istiod-ca-cert\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.299626 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.299603 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5mq2\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-kube-api-access-l5mq2\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.299672 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.299657 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-credential-socket\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.299716 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.299695 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-token\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.299753 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.299731 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-podinfo\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.299796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.299756 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-data\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.299796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.299782 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-certs\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.299893 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.299806 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-envoy\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.400773 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.400741 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istiod-ca-cert\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.400773 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.400775 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5mq2\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-kube-api-access-l5mq2\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401027 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.400801 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-credential-socket\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401027 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.400822 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-token\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401027 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.400844 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-podinfo\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.401077 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-data\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.401121 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-certs\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.401145 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-envoy\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.401188 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-socket\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.401222 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-credential-socket\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.401421 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-data\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.401574 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-certs\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.401574 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istiod-ca-cert\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.401643 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.401630 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-socket\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.403365 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.403347 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-envoy\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.403445 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.403403 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-podinfo\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.407812 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.407782 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-token\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.408001 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.407985 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5mq2\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-kube-api-access-l5mq2\") pod \"e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.575763 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.575669 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:17.713562 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.713538 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn"] Aug 04 00:49:17.715661 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:49:17.715627 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50785fa2_c4da_4c32_9a36_1f2e36396f7f.slice/crio-4e881b83ada7970ca964f152e7de611ba8d2ab25ffbfa4a2c05ef8cb2e4bdc72 WatchSource:0}: Error finding container 4e881b83ada7970ca964f152e7de611ba8d2ab25ffbfa4a2c05ef8cb2e4bdc72: Status 404 returned error can't find the container with id 4e881b83ada7970ca964f152e7de611ba8d2ab25ffbfa4a2c05ef8cb2e4bdc72 Aug 04 00:49:17.718118 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.718088 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:49:17.718204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.718154 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:49:17.718204 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.718183 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:49:17.791428 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.791394 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" event={"ID":"50785fa2-c4da-4c32-9a36-1f2e36396f7f","Type":"ContainerStarted","Data":"c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c"} Aug 04 00:49:17.791597 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.791447 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" event={"ID":"50785fa2-c4da-4c32-9a36-1f2e36396f7f","Type":"ContainerStarted","Data":"4e881b83ada7970ca964f152e7de611ba8d2ab25ffbfa4a2c05ef8cb2e4bdc72"} Aug 04 00:49:17.811110 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:17.811049 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" podStartSLOduration=0.811022211 podStartE2EDuration="811.022211ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:49:17.809618897 +0000 UTC m=+2796.472528376" watchObservedRunningTime="2026-08-04 00:49:17.811022211 +0000 UTC m=+2796.473931690" Aug 04 00:49:18.576650 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:18.576612 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:18.581720 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:18.581693 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:18.796582 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:18.796554 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:18.797623 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:18.797600 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:21.417516 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.417483 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464"] Aug 04 00:49:21.417987 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.417950 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df2ac31-792a-47ab-b352-942ad9cdb3c4" containerName="istio-proxy" Aug 04 00:49:21.420867 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.420836 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:21.423185 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.423165 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5782b0\"" Aug 04 00:49:21.423334 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.423314 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5782b0-dockercfg-8lq7z\"" Aug 04 00:49:21.431661 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.431632 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464"] Aug 04 00:49:21.445806 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.445775 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2cb2\" (UniqueName: \"kubernetes.io/projected/8b395a08-ca50-4787-97ea-cb12e0107c2e-kube-api-access-r2cb2\") pod \"payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464\" (UID: \"8b395a08-ca50-4787-97ea-cb12e0107c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:21.445952 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.445879 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b395a08-ca50-4787-97ea-cb12e0107c2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464\" (UID: \"8b395a08-ca50-4787-97ea-cb12e0107c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:21.547518 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.547467 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2cb2\" (UniqueName: \"kubernetes.io/projected/8b395a08-ca50-4787-97ea-cb12e0107c2e-kube-api-access-r2cb2\") pod \"payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464\" (UID: \"8b395a08-ca50-4787-97ea-cb12e0107c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:21.547691 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.547550 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b395a08-ca50-4787-97ea-cb12e0107c2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464\" (UID: \"8b395a08-ca50-4787-97ea-cb12e0107c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:21.548520 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.548492 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b395a08-ca50-4787-97ea-cb12e0107c2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464\" (UID: \"8b395a08-ca50-4787-97ea-cb12e0107c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:21.557029 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.557004 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2cb2\" (UniqueName: \"kubernetes.io/projected/8b395a08-ca50-4787-97ea-cb12e0107c2e-kube-api-access-r2cb2\") pod \"payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464\" (UID: \"8b395a08-ca50-4787-97ea-cb12e0107c2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:21.733398 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:21.733309 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:22.075836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:22.075807 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464"] Aug 04 00:49:22.077479 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:49:22.077449 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b395a08_ca50_4787_97ea_cb12e0107c2e.slice/crio-20a8bc299e83326a72e1dd4cec56603ae08a47397a290058b2d386120c6b83ef WatchSource:0}: Error finding container 20a8bc299e83326a72e1dd4cec56603ae08a47397a290058b2d386120c6b83ef: Status 404 returned error can't find the container with id 20a8bc299e83326a72e1dd4cec56603ae08a47397a290058b2d386120c6b83ef Aug 04 00:49:22.820388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:22.820353 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" event={"ID":"8b395a08-ca50-4787-97ea-cb12e0107c2e","Type":"ContainerStarted","Data":"0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e"} Aug 04 00:49:22.820785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:22.820406 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" event={"ID":"8b395a08-ca50-4787-97ea-cb12e0107c2e","Type":"ContainerStarted","Data":"20a8bc299e83326a72e1dd4cec56603ae08a47397a290058b2d386120c6b83ef"} Aug 04 00:49:22.820785 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:22.820484 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:22.837302 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:22.837250 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" podStartSLOduration=1.837235929 podStartE2EDuration="1.837235929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:49:22.834596861 +0000 UTC m=+2801.497506342" watchObservedRunningTime="2026-08-04 00:49:22.837235929 +0000 UTC m=+2801.500145452" Aug 04 00:49:33.798082 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:33.798047 2552 patch_prober.go:28] interesting pod/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Aug 04 00:49:33.798543 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:33.798119 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Aug 04 00:49:33.827525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:33.827497 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:49:33.877637 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:33.877609 2552 generic.go:356] "Generic (PLEG): container finished" podID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerID="c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c" exitCode=0 Aug 04 00:49:33.877778 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:33.877664 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" event={"ID":"50785fa2-c4da-4c32-9a36-1f2e36396f7f","Type":"ContainerDied","Data":"c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c"} Aug 04 00:49:33.878061 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:33.878045 2552 scope.go:117] "RemoveContainer" containerID="c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c" Aug 04 00:49:33.879290 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:33.879253 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:49:33.879361 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:33.879350 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:49:33.879402 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:33.879385 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:49:34.888605 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:34.888557 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" event={"ID":"50785fa2-c4da-4c32-9a36-1f2e36396f7f","Type":"ContainerStarted","Data":"e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa"} Aug 04 00:49:35.576186 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:35.576151 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:35.576402 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:35.576288 2552 patch_prober.go:28] interesting pod/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Aug 04 00:49:35.576402 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:35.576343 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Aug 04 00:49:36.576718 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:36.576675 2552 patch_prober.go:28] interesting pod/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Aug 04 00:49:36.577344 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:36.577312 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Aug 04 00:49:37.576738 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:37.576695 2552 patch_prober.go:28] interesting pod/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Aug 04 00:49:37.577346 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:37.577312 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Aug 04 00:49:38.576660 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:38.576615 2552 patch_prober.go:28] interesting pod/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Aug 04 00:49:38.576887 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:38.576686 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Aug 04 00:49:39.577116 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:39.577068 2552 patch_prober.go:28] interesting pod/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Aug 04 00:49:39.577620 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:39.577139 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Aug 04 00:49:40.580278 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:40.580251 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:40.580779 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:40.580766 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:49:40.581431 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:49:40.581412 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:50:09.479196 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:09.479157 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf"] Aug 04 00:50:09.479685 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:09.479358 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" containerID="cri-o://3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4" gracePeriod=30 Aug 04 00:50:11.155106 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:11.155065 2552 patch_prober.go:28] interesting pod/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 00:50:11.155484 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:11.155143 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:50:14.729105 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.729080 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:50:14.857170 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857086 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tttr4\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-kube-api-access-tttr4\") pod \"aec0d949-090a-496c-8965-55c5df56b9f4\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " Aug 04 00:50:14.857170 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857129 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec0d949-090a-496c-8965-55c5df56b9f4-istiod-ca-cert\") pod \"aec0d949-090a-496c-8965-55c5df56b9f4\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " Aug 04 00:50:14.857170 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857163 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-istio-token\") pod \"aec0d949-090a-496c-8965-55c5df56b9f4\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " Aug 04 00:50:14.857437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857188 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec0d949-090a-496c-8965-55c5df56b9f4-istio-podinfo\") pod \"aec0d949-090a-496c-8965-55c5df56b9f4\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " Aug 04 00:50:14.857437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857240 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-socket\") pod \"aec0d949-090a-496c-8965-55c5df56b9f4\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " Aug 04 00:50:14.857437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857274 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-envoy\") pod \"aec0d949-090a-496c-8965-55c5df56b9f4\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " Aug 04 00:50:14.857437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857293 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-certs\") pod \"aec0d949-090a-496c-8965-55c5df56b9f4\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " Aug 04 00:50:14.857437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857335 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-data\") pod \"aec0d949-090a-496c-8965-55c5df56b9f4\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " Aug 04 00:50:14.857437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857355 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-credential-socket\") pod \"aec0d949-090a-496c-8965-55c5df56b9f4\" (UID: \"aec0d949-090a-496c-8965-55c5df56b9f4\") " Aug 04 00:50:14.857735 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857566 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec0d949-090a-496c-8965-55c5df56b9f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aec0d949-090a-496c-8965-55c5df56b9f4" (UID: "aec0d949-090a-496c-8965-55c5df56b9f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:50:14.857800 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857705 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aec0d949-090a-496c-8965-55c5df56b9f4" (UID: "aec0d949-090a-496c-8965-55c5df56b9f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:14.857800 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857785 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aec0d949-090a-496c-8965-55c5df56b9f4" (UID: "aec0d949-090a-496c-8965-55c5df56b9f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:14.857944 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857801 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aec0d949-090a-496c-8965-55c5df56b9f4" (UID: "aec0d949-090a-496c-8965-55c5df56b9f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:14.857944 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857809 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec0d949-090a-496c-8965-55c5df56b9f4-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:14.858070 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.857939 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "aec0d949-090a-496c-8965-55c5df56b9f4" (UID: "aec0d949-090a-496c-8965-55c5df56b9f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:14.859423 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.859391 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "aec0d949-090a-496c-8965-55c5df56b9f4" (UID: "aec0d949-090a-496c-8965-55c5df56b9f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:14.859423 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.859393 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aec0d949-090a-496c-8965-55c5df56b9f4" (UID: "aec0d949-090a-496c-8965-55c5df56b9f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:14.859573 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.859427 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aec0d949-090a-496c-8965-55c5df56b9f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aec0d949-090a-496c-8965-55c5df56b9f4" (UID: "aec0d949-090a-496c-8965-55c5df56b9f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:50:14.859615 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.859596 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-kube-api-access-tttr4" (OuterVolumeSpecName: "kube-api-access-tttr4") pod "aec0d949-090a-496c-8965-55c5df56b9f4" (UID: "aec0d949-090a-496c-8965-55c5df56b9f4"). InnerVolumeSpecName "kube-api-access-tttr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:14.958791 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.958761 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tttr4\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-kube-api-access-tttr4\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:14.958791 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.958788 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec0d949-090a-496c-8965-55c5df56b9f4-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:14.958984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.958801 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec0d949-090a-496c-8965-55c5df56b9f4-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:14.958984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.958812 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:14.958984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.958824 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:14.958984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.958836 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:14.958984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.958847 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:14.958984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:14.958882 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec0d949-090a-496c-8965-55c5df56b9f4-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:15.086397 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.086353 2552 generic.go:356] "Generic (PLEG): container finished" podID="aec0d949-090a-496c-8965-55c5df56b9f4" containerID="3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4" exitCode=0 Aug 04 00:50:15.086554 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.086454 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" Aug 04 00:50:15.086554 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.086491 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" event={"ID":"aec0d949-090a-496c-8965-55c5df56b9f4","Type":"ContainerDied","Data":"3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4"} Aug 04 00:50:15.086554 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.086545 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf" event={"ID":"aec0d949-090a-496c-8965-55c5df56b9f4","Type":"ContainerDied","Data":"2b4d4517d94bfd6d01e6ef7be34d10a1fd58fb3f92f31ab8a4cdbe7b766c3354"} Aug 04 00:50:15.086661 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.086566 2552 scope.go:117] "RemoveContainer" containerID="3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4" Aug 04 00:50:15.096797 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.096779 2552 scope.go:117] "RemoveContainer" containerID="49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206" Aug 04 00:50:15.107617 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.107594 2552 scope.go:117] "RemoveContainer" containerID="3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4" Aug 04 00:50:15.108014 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:50:15.107882 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4\": container with ID starting with 3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4 not found: ID does not exist" containerID="3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4" Aug 04 00:50:15.108014 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.107917 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4"} err="failed to get container status \"3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4\": rpc error: code = NotFound desc = could not find container \"3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4\": container with ID starting with 3ee94ed990118280bffc9e2045ff3d11b4e9a0ce6ba6b821fc3bd5a6238604c4 not found: ID does not exist" Aug 04 00:50:15.108014 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.107941 2552 scope.go:117] "RemoveContainer" containerID="49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206" Aug 04 00:50:15.108246 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:50:15.108223 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206\": container with ID starting with 49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206 not found: ID does not exist" containerID="49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206" Aug 04 00:50:15.108300 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.108255 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206"} err="failed to get container status \"49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206\": rpc error: code = NotFound desc = could not find container \"49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206\": container with ID starting with 49046fea36178e5517bbffd7e07fe806f6a86dc91fc9bdb8785f05c40117b206 not found: ID does not exist" Aug 04 00:50:15.109202 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.109183 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf"] Aug 04 00:50:15.112935 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.112913 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a12ead-openshift-default-5b6f9fc46d-2wmpf"] Aug 04 00:50:15.953640 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:15.953609 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" path="/var/lib/kubelet/pods/aec0d949-090a-496c-8965-55c5df56b9f4/volumes" Aug 04 00:50:25.344669 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:25.344621 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464"] Aug 04 00:50:25.345207 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:25.344944 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" podUID="8b395a08-ca50-4787-97ea-cb12e0107c2e" containerName="payload-pre-processing" containerID="cri-o://0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e" gracePeriod=30 Aug 04 00:50:25.598428 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:25.598373 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:50:25.750128 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:25.750098 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2cb2\" (UniqueName: \"kubernetes.io/projected/8b395a08-ca50-4787-97ea-cb12e0107c2e-kube-api-access-r2cb2\") pod \"8b395a08-ca50-4787-97ea-cb12e0107c2e\" (UID: \"8b395a08-ca50-4787-97ea-cb12e0107c2e\") " Aug 04 00:50:25.750303 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:25.750219 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b395a08-ca50-4787-97ea-cb12e0107c2e-plugins-config-volume\") pod \"8b395a08-ca50-4787-97ea-cb12e0107c2e\" (UID: \"8b395a08-ca50-4787-97ea-cb12e0107c2e\") " Aug 04 00:50:25.750590 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:25.750566 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b395a08-ca50-4787-97ea-cb12e0107c2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b395a08-ca50-4787-97ea-cb12e0107c2e" (UID: "8b395a08-ca50-4787-97ea-cb12e0107c2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:50:25.752245 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:25.752220 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b395a08-ca50-4787-97ea-cb12e0107c2e-kube-api-access-r2cb2" (OuterVolumeSpecName: "kube-api-access-r2cb2") pod "8b395a08-ca50-4787-97ea-cb12e0107c2e" (UID: "8b395a08-ca50-4787-97ea-cb12e0107c2e"). InnerVolumeSpecName "kube-api-access-r2cb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:25.851682 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:25.851613 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2cb2\" (UniqueName: \"kubernetes.io/projected/8b395a08-ca50-4787-97ea-cb12e0107c2e-kube-api-access-r2cb2\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:25.851682 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:25.851636 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b395a08-ca50-4787-97ea-cb12e0107c2e-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:26.144517 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:26.144484 2552 generic.go:356] "Generic (PLEG): container finished" podID="8b395a08-ca50-4787-97ea-cb12e0107c2e" containerID="0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e" exitCode=0 Aug 04 00:50:26.144700 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:26.144540 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" Aug 04 00:50:26.144700 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:26.144574 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" event={"ID":"8b395a08-ca50-4787-97ea-cb12e0107c2e","Type":"ContainerDied","Data":"0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e"} Aug 04 00:50:26.144700 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:26.144632 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464" event={"ID":"8b395a08-ca50-4787-97ea-cb12e0107c2e","Type":"ContainerDied","Data":"20a8bc299e83326a72e1dd4cec56603ae08a47397a290058b2d386120c6b83ef"} Aug 04 00:50:26.144700 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:26.144655 2552 scope.go:117] "RemoveContainer" containerID="0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e" Aug 04 00:50:26.153188 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:26.153172 2552 scope.go:117] "RemoveContainer" containerID="0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e" Aug 04 00:50:26.153420 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:50:26.153400 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e\": container with ID starting with 0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e not found: ID does not exist" containerID="0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e" Aug 04 00:50:26.153464 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:26.153427 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e"} err="failed to get container status \"0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e\": rpc error: code = NotFound desc = could not find container \"0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e\": container with ID starting with 0bf34c1b6854e633f4c4e2fc2b3016e1b20c4bdf27cf24768f8ae6844e18785e not found: ID does not exist" Aug 04 00:50:26.159928 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:26.159906 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464"] Aug 04 00:50:26.163671 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:26.163651 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5782b0-86f8874bb8-zg464"] Aug 04 00:50:27.953080 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:27.953048 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b395a08-ca50-4787-97ea-cb12e0107c2e" path="/var/lib/kubelet/pods/8b395a08-ca50-4787-97ea-cb12e0107c2e/volumes" Aug 04 00:50:31.929421 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:31.929390 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn"] Aug 04 00:50:31.929811 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:31.929597 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" containerID="cri-o://e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa" gracePeriod=30 Aug 04 00:50:32.806525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.806487 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm"] Aug 04 00:50:32.807110 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.807086 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" Aug 04 00:50:32.807187 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.807113 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" Aug 04 00:50:32.807187 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.807163 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" Aug 04 00:50:32.807187 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.807174 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" Aug 04 00:50:32.807283 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.807185 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b395a08-ca50-4787-97ea-cb12e0107c2e" containerName="payload-pre-processing" Aug 04 00:50:32.807283 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.807195 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b395a08-ca50-4787-97ea-cb12e0107c2e" containerName="payload-pre-processing" Aug 04 00:50:32.807349 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.807292 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" Aug 04 00:50:32.807349 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.807307 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b395a08-ca50-4787-97ea-cb12e0107c2e" containerName="payload-pre-processing" Aug 04 00:50:32.807349 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.807318 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec0d949-090a-496c-8965-55c5df56b9f4" containerName="istio-proxy" Aug 04 00:50:32.810459 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.810438 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:32.812899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.812868 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-5ddf6e-openshift-default-dockercfg-t6sfs\"" Aug 04 00:50:32.820523 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.820502 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm"] Aug 04 00:50:32.913454 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.913424 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jff7\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-kube-api-access-7jff7\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:32.913631 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.913462 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-data\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:32.913631 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.913504 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-certs\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:32.913631 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.913575 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-podinfo\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:32.913750 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.913638 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed002a2-9025-4a51-9cbb-6919bcb967a3-istiod-ca-cert\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:32.913750 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.913697 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-envoy\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:32.913750 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.913732 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-socket\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:32.913847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.913765 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-credential-socket\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:32.913847 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:32.913815 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-token\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.014630 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.014588 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-credential-socket\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015147 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.014641 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-token\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015147 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.014689 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jff7\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-kube-api-access-7jff7\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015147 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.014724 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-data\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015147 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.014799 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-certs\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015147 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.014825 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-podinfo\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015147 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.014848 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed002a2-9025-4a51-9cbb-6919bcb967a3-istiod-ca-cert\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015623 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.015587 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-envoy\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015761 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.015626 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed002a2-9025-4a51-9cbb-6919bcb967a3-istiod-ca-cert\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015761 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.015636 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-socket\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015761 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.015042 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-credential-socket\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015761 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.015123 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-data\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.015761 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.015215 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-certs\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.016092 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.015845 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-socket\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.017503 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.017481 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-podinfo\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.017616 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.017581 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-envoy\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.023179 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.023157 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-token\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.023262 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.023209 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jff7\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-kube-api-access-7jff7\") pod \"e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.121355 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.121326 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:33.260732 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.260703 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm"] Aug 04 00:50:33.262486 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:50:33.262458 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed002a2_9025_4a51_9cbb_6919bcb967a3.slice/crio-4f2098ad34565274e799e9195b24c690a88ba90f21663b0b0d6a9a7b34f446e0 WatchSource:0}: Error finding container 4f2098ad34565274e799e9195b24c690a88ba90f21663b0b0d6a9a7b34f446e0: Status 404 returned error can't find the container with id 4f2098ad34565274e799e9195b24c690a88ba90f21663b0b0d6a9a7b34f446e0 Aug 04 00:50:33.264696 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.264664 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:50:33.264796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.264725 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:50:33.264796 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:33.264754 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:50:34.190443 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:34.190403 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" event={"ID":"eed002a2-9025-4a51-9cbb-6919bcb967a3","Type":"ContainerStarted","Data":"10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd"} Aug 04 00:50:34.190900 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:34.190480 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" event={"ID":"eed002a2-9025-4a51-9cbb-6919bcb967a3","Type":"ContainerStarted","Data":"4f2098ad34565274e799e9195b24c690a88ba90f21663b0b0d6a9a7b34f446e0"} Aug 04 00:50:34.210503 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:34.210453 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" podStartSLOduration=2.210439143 podStartE2EDuration="2.210439143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:50:34.209111204 +0000 UTC m=+2872.872020685" watchObservedRunningTime="2026-08-04 00:50:34.210439143 +0000 UTC m=+2872.873348623" Aug 04 00:50:35.121841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:35.121812 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:35.126771 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:35.126746 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:35.195437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:35.195405 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:35.196452 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:35.196428 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:37.189143 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.189118 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:50:37.207548 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.207470 2552 generic.go:356] "Generic (PLEG): container finished" podID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerID="e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa" exitCode=0 Aug 04 00:50:37.207548 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.207508 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" event={"ID":"50785fa2-c4da-4c32-9a36-1f2e36396f7f","Type":"ContainerDied","Data":"e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa"} Aug 04 00:50:37.207719 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.207579 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" event={"ID":"50785fa2-c4da-4c32-9a36-1f2e36396f7f","Type":"ContainerDied","Data":"4e881b83ada7970ca964f152e7de611ba8d2ab25ffbfa4a2c05ef8cb2e4bdc72"} Aug 04 00:50:37.207719 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.207521 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn" Aug 04 00:50:37.207719 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.207614 2552 scope.go:117] "RemoveContainer" containerID="e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa" Aug 04 00:50:37.218205 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.218186 2552 scope.go:117] "RemoveContainer" containerID="c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c" Aug 04 00:50:37.226054 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.226039 2552 scope.go:117] "RemoveContainer" containerID="e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa" Aug 04 00:50:37.226336 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:50:37.226313 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa\": container with ID starting with e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa not found: ID does not exist" containerID="e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa" Aug 04 00:50:37.226422 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.226341 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa"} err="failed to get container status \"e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa\": rpc error: code = NotFound desc = could not find container \"e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa\": container with ID starting with e773388204fa3094f035c30b393654e0b7daedf8a312bd4afc68a4703eb4abfa not found: ID does not exist" Aug 04 00:50:37.226422 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.226357 2552 scope.go:117] "RemoveContainer" containerID="c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c" Aug 04 00:50:37.226619 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:50:37.226603 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c\": container with ID starting with c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c not found: ID does not exist" containerID="c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c" Aug 04 00:50:37.226697 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.226624 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c"} err="failed to get container status \"c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c\": rpc error: code = NotFound desc = could not find container \"c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c\": container with ID starting with c032ec235eee361dbf0f6467185dc80143faa875223c4cad2322833b2a7ba96c not found: ID does not exist" Aug 04 00:50:37.354028 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.353988 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-certs\") pod \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " Aug 04 00:50:37.354028 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354042 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istiod-ca-cert\") pod \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " Aug 04 00:50:37.354267 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354064 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-credential-socket\") pod \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " Aug 04 00:50:37.354267 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354090 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-envoy\") pod \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " Aug 04 00:50:37.354267 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354124 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5mq2\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-kube-api-access-l5mq2\") pod \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " Aug 04 00:50:37.354267 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354148 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-token\") pod \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " Aug 04 00:50:37.354267 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354176 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-data\") pod \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " Aug 04 00:50:37.354267 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354222 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-socket\") pod \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " Aug 04 00:50:37.354590 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354281 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-podinfo\") pod \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\" (UID: \"50785fa2-c4da-4c32-9a36-1f2e36396f7f\") " Aug 04 00:50:37.354590 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354312 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50785fa2-c4da-4c32-9a36-1f2e36396f7f" (UID: "50785fa2-c4da-4c32-9a36-1f2e36396f7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:37.354590 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354390 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50785fa2-c4da-4c32-9a36-1f2e36396f7f" (UID: "50785fa2-c4da-4c32-9a36-1f2e36396f7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:37.354757 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354606 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:37.354757 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354629 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:37.354757 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354602 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50785fa2-c4da-4c32-9a36-1f2e36396f7f" (UID: "50785fa2-c4da-4c32-9a36-1f2e36396f7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:37.354757 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354618 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-data" (OuterVolumeSpecName: "istio-data") pod "50785fa2-c4da-4c32-9a36-1f2e36396f7f" (UID: "50785fa2-c4da-4c32-9a36-1f2e36396f7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:37.354757 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.354676 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50785fa2-c4da-4c32-9a36-1f2e36396f7f" (UID: "50785fa2-c4da-4c32-9a36-1f2e36396f7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:50:37.356558 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.356527 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-token" (OuterVolumeSpecName: "istio-token") pod "50785fa2-c4da-4c32-9a36-1f2e36396f7f" (UID: "50785fa2-c4da-4c32-9a36-1f2e36396f7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:37.356558 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.356546 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-kube-api-access-l5mq2" (OuterVolumeSpecName: "kube-api-access-l5mq2") pod "50785fa2-c4da-4c32-9a36-1f2e36396f7f" (UID: "50785fa2-c4da-4c32-9a36-1f2e36396f7f"). InnerVolumeSpecName "kube-api-access-l5mq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:37.356710 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.356557 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50785fa2-c4da-4c32-9a36-1f2e36396f7f" (UID: "50785fa2-c4da-4c32-9a36-1f2e36396f7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 00:50:37.356710 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.356551 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50785fa2-c4da-4c32-9a36-1f2e36396f7f" (UID: "50785fa2-c4da-4c32-9a36-1f2e36396f7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:50:37.455731 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.455689 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:37.455731 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.455726 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:37.455731 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.455735 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5mq2\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-kube-api-access-l5mq2\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:37.455731 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.455743 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:37.456014 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.455752 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:37.456014 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.455759 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50785fa2-c4da-4c32-9a36-1f2e36396f7f-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:37.456014 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.455766 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50785fa2-c4da-4c32-9a36-1f2e36396f7f-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:50:37.531439 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.531410 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn"] Aug 04 00:50:37.535542 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.535521 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5782b0-openshift-default-5b89c496db-pwlfn"] Aug 04 00:50:37.956525 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:37.956492 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" path="/var/lib/kubelet/pods/50785fa2-c4da-4c32-9a36-1f2e36396f7f/volumes" Aug 04 00:50:50.196802 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:50.196759 2552 patch_prober.go:28] interesting pod/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Aug 04 00:50:50.197269 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:50.196831 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Aug 04 00:50:50.274347 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:50.274313 2552 generic.go:356] "Generic (PLEG): container finished" podID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerID="10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd" exitCode=0 Aug 04 00:50:50.274531 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:50.274378 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" event={"ID":"eed002a2-9025-4a51-9cbb-6919bcb967a3","Type":"ContainerDied","Data":"10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd"} Aug 04 00:50:50.274744 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:50.274731 2552 scope.go:117] "RemoveContainer" containerID="10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd" Aug 04 00:50:50.275843 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:50.275811 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:50:50.275943 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:50.275926 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:50:50.275994 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:50.275967 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:50:51.306893 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.306792 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" event={"ID":"eed002a2-9025-4a51-9cbb-6919bcb967a3","Type":"ContainerStarted","Data":"49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29"} Aug 04 00:50:51.381913 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.379842 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj"] Aug 04 00:50:51.381913 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.380370 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" Aug 04 00:50:51.381913 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.380389 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" Aug 04 00:50:51.381913 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.380411 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" Aug 04 00:50:51.381913 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.380419 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" Aug 04 00:50:51.381913 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.380527 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" Aug 04 00:50:51.384647 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.384618 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.390684 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.388905 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-33bda8-openshift-default-dockercfg-m5spq\"" Aug 04 00:50:51.398752 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.398722 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj"] Aug 04 00:50:51.475570 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.475531 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-istio-token\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.475846 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.475826 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9dac14-724b-473c-91fd-c294923dd847-istio-podinfo\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.476135 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.476114 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-certs\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.476260 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.476242 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-data\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.476413 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.476397 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-socket\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.476516 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.476503 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-envoy\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.476627 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.476612 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95tbc\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-kube-api-access-95tbc\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.476735 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.476720 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-credential-socket\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.476841 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.476825 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9dac14-724b-473c-91fd-c294923dd847-istiod-ca-cert\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.577662 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.577566 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95tbc\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-kube-api-access-95tbc\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.577662 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.577620 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-credential-socket\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.577662 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.577657 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9dac14-724b-473c-91fd-c294923dd847-istiod-ca-cert\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.577986 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.577690 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-istio-token\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.577986 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.577723 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9dac14-724b-473c-91fd-c294923dd847-istio-podinfo\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.577986 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.577792 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-certs\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.577986 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.577817 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-data\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.577986 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.577885 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-socket\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.577986 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.577914 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-envoy\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.578720 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.578693 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-credential-socket\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.579199 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.579146 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-certs\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.579343 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.579229 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-data\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.579458 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.579337 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9dac14-724b-473c-91fd-c294923dd847-istiod-ca-cert\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.579557 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.579497 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-socket\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.582942 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.582915 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9dac14-724b-473c-91fd-c294923dd847-istio-podinfo\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.583085 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.583061 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-envoy\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.588940 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.588917 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95tbc\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-kube-api-access-95tbc\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.590308 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.590287 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-istio-token\") pod \"e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.712881 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.712826 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:51.893127 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.882526 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj"] Aug 04 00:50:51.906135 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.906102 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:50:51.906213 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.906186 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:50:51.906254 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:51.906217 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:50:52.121550 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:52.121506 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:52.121762 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:52.121734 2552 patch_prober.go:28] interesting pod/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Aug 04 00:50:52.121822 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:52.121798 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Aug 04 00:50:52.315237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:52.315157 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" event={"ID":"6b9dac14-724b-473c-91fd-c294923dd847","Type":"ContainerStarted","Data":"e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d"} Aug 04 00:50:52.315237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:52.315209 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" event={"ID":"6b9dac14-724b-473c-91fd-c294923dd847","Type":"ContainerStarted","Data":"54f9b895bf3924be95f39a7bdc310e15dd9afb304418bc95c38334f64e85fabf"} Aug 04 00:50:52.334848 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:52.334794 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" podStartSLOduration=1.334776782 podStartE2EDuration="1.334776782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:50:52.334271695 +0000 UTC m=+2890.997181175" watchObservedRunningTime="2026-08-04 00:50:52.334776782 +0000 UTC m=+2890.997686260" Aug 04 00:50:52.713034 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:52.712996 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:52.718218 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:52.718194 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:53.122396 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:53.122359 2552 patch_prober.go:28] interesting pod/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Aug 04 00:50:53.122560 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:53.122431 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Aug 04 00:50:53.319836 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:53.319800 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:53.320740 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:53.320720 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:50:54.122275 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:54.122234 2552 patch_prober.go:28] interesting pod/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Aug 04 00:50:54.122467 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:54.122304 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Aug 04 00:50:55.122218 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:55.122187 2552 patch_prober.go:28] interesting pod/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Aug 04 00:50:55.122586 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:55.122244 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Aug 04 00:50:56.122041 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:56.122008 2552 patch_prober.go:28] interesting pod/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Aug 04 00:50:56.122219 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:56.122063 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Aug 04 00:50:57.125699 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:57.125661 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:57.126180 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:57.125993 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:50:57.126769 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:50:57.126750 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 00:51:08.477318 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:08.476453 2552 patch_prober.go:28] interesting pod/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": read tcp 10.133.0.2:48032->10.133.0.67:15021: read: connection reset by peer" start-of-body= Aug 04 00:51:08.477318 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:08.476538 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": read tcp 10.133.0.2:48032->10.133.0.67:15021: read: connection reset by peer" Aug 04 00:51:09.403489 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:09.403456 2552 generic.go:356] "Generic (PLEG): container finished" podID="6b9dac14-724b-473c-91fd-c294923dd847" containerID="e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d" exitCode=0 Aug 04 00:51:09.403672 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:09.403518 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" event={"ID":"6b9dac14-724b-473c-91fd-c294923dd847","Type":"ContainerDied","Data":"e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d"} Aug 04 00:51:09.403897 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:09.403883 2552 scope.go:117] "RemoveContainer" containerID="e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d" Aug 04 00:51:09.404908 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:09.404874 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:51:09.405031 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:09.404966 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:51:09.405031 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:09.405013 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 00:51:10.409984 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:10.409945 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" event={"ID":"6b9dac14-724b-473c-91fd-c294923dd847","Type":"ContainerStarted","Data":"8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda"} Aug 04 00:51:10.713812 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:10.713722 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:51:10.713992 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:10.713906 2552 patch_prober.go:28] interesting pod/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Aug 04 00:51:10.713992 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:10.713966 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Aug 04 00:51:11.713438 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:11.713403 2552 patch_prober.go:28] interesting pod/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Aug 04 00:51:11.713843 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:11.713460 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Aug 04 00:51:12.713449 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:12.713415 2552 patch_prober.go:28] interesting pod/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Aug 04 00:51:12.713909 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:12.713473 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Aug 04 00:51:13.713710 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:13.713674 2552 patch_prober.go:28] interesting pod/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Aug 04 00:51:13.714145 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:13.713737 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Aug 04 00:51:14.713469 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:14.713428 2552 patch_prober.go:28] interesting pod/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Aug 04 00:51:14.713669 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:14.713485 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Aug 04 00:51:15.716612 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:15.716580 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:51:15.717069 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:15.716964 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:51:15.717702 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:15.717686 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 00:51:46.721830 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.721792 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2"] Aug 04 00:51:46.722690 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.722562 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="50785fa2-c4da-4c32-9a36-1f2e36396f7f" containerName="istio-proxy" Aug 04 00:51:46.725795 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.725772 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.728389 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.728368 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"default-dockercfg-5qnwp\"" Aug 04 00:51:46.729406 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.729387 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"openshift-service-ca.crt\"" Aug 04 00:51:46.729510 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.729404 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"auth-test-model-33bda8-kserve-self-signed-certs\"" Aug 04 00:51:46.729510 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.729388 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"kube-root-ca.crt\"" Aug 04 00:51:46.735205 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.734368 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2"] Aug 04 00:51:46.791583 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.791545 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr54b\" (UniqueName: \"kubernetes.io/projected/28c9d555-a3c9-4232-a340-50b47283d82c-kube-api-access-lr54b\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.791583 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.791584 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-home\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.791791 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.791601 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-model-cache\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.791791 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.791616 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-tmp-dir\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.791791 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.791703 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c9d555-a3c9-4232-a340-50b47283d82c-tls-certs\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.791938 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.791796 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-dshm\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.892271 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.892236 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-dshm\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.892456 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.892297 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr54b\" (UniqueName: \"kubernetes.io/projected/28c9d555-a3c9-4232-a340-50b47283d82c-kube-api-access-lr54b\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.892456 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.892315 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-home\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.892456 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.892329 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-model-cache\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.892456 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.892345 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-tmp-dir\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.892456 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.892370 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c9d555-a3c9-4232-a340-50b47283d82c-tls-certs\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.892758 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.892728 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-home\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.892813 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.892765 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-tmp-dir\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.892813 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.892794 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-model-cache\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.894518 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.894492 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-dshm\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.894751 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.894734 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c9d555-a3c9-4232-a340-50b47283d82c-tls-certs\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:46.900394 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:46.900372 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr54b\" (UniqueName: \"kubernetes.io/projected/28c9d555-a3c9-4232-a340-50b47283d82c-kube-api-access-lr54b\") pod \"auth-test-model-33bda8-kserve-c67d66987-z8zg2\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:47.046470 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:47.046383 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:47.183486 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:47.183451 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2"] Aug 04 00:51:47.185230 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:51:47.185208 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28c9d555_a3c9_4232_a340_50b47283d82c.slice/crio-7012cc9886d539ed15b7cbd0ded5b9dfea93ecef881e421f5791e2da42af37b5 WatchSource:0}: Error finding container 7012cc9886d539ed15b7cbd0ded5b9dfea93ecef881e421f5791e2da42af37b5: Status 404 returned error can't find the container with id 7012cc9886d539ed15b7cbd0ded5b9dfea93ecef881e421f5791e2da42af37b5 Aug 04 00:51:47.598372 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:47.598333 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" event={"ID":"28c9d555-a3c9-4232-a340-50b47283d82c","Type":"ContainerStarted","Data":"3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc"} Aug 04 00:51:47.598372 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:47.598375 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" event={"ID":"28c9d555-a3c9-4232-a340-50b47283d82c","Type":"ContainerStarted","Data":"7012cc9886d539ed15b7cbd0ded5b9dfea93ecef881e421f5791e2da42af37b5"} Aug 04 00:51:47.613768 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:47.613719 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" podStartSLOduration=1.6137047199999999 podStartE2EDuration="1.61370472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:51:47.613010726 +0000 UTC m=+2946.275920205" watchObservedRunningTime="2026-08-04 00:51:47.61370472 +0000 UTC m=+2946.276614201" Aug 04 00:51:57.047069 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:57.047027 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:57.047516 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:57.047118 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:57.059442 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:57.059420 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:51:57.659229 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:51:57.659199 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:55:56.843189 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:55:56.843142 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2"] Aug 04 00:56:11.862531 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:11.862480 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" podUID="28c9d555-a3c9-4232-a340-50b47283d82c" containerName="main" containerID="cri-o://3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc" gracePeriod=45 Aug 04 00:56:12.116348 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.116283 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:56:12.196055 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196022 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr54b\" (UniqueName: \"kubernetes.io/projected/28c9d555-a3c9-4232-a340-50b47283d82c-kube-api-access-lr54b\") pod \"28c9d555-a3c9-4232-a340-50b47283d82c\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " Aug 04 00:56:12.196236 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196073 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-model-cache\") pod \"28c9d555-a3c9-4232-a340-50b47283d82c\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " Aug 04 00:56:12.196236 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196144 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c9d555-a3c9-4232-a340-50b47283d82c-tls-certs\") pod \"28c9d555-a3c9-4232-a340-50b47283d82c\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " Aug 04 00:56:12.196236 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196220 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-tmp-dir\") pod \"28c9d555-a3c9-4232-a340-50b47283d82c\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " Aug 04 00:56:12.196418 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196244 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-dshm\") pod \"28c9d555-a3c9-4232-a340-50b47283d82c\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " Aug 04 00:56:12.196418 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196261 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-home\") pod \"28c9d555-a3c9-4232-a340-50b47283d82c\" (UID: \"28c9d555-a3c9-4232-a340-50b47283d82c\") " Aug 04 00:56:12.196418 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196282 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-model-cache" (OuterVolumeSpecName: "model-cache") pod "28c9d555-a3c9-4232-a340-50b47283d82c" (UID: "28c9d555-a3c9-4232-a340-50b47283d82c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:56:12.196586 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196471 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28c9d555-a3c9-4232-a340-50b47283d82c" (UID: "28c9d555-a3c9-4232-a340-50b47283d82c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:56:12.196586 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196544 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-home" (OuterVolumeSpecName: "home") pod "28c9d555-a3c9-4232-a340-50b47283d82c" (UID: "28c9d555-a3c9-4232-a340-50b47283d82c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:56:12.196699 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196606 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:56:12.196699 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196617 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:56:12.196699 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.196625 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:56:12.198342 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.198311 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c9d555-a3c9-4232-a340-50b47283d82c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28c9d555-a3c9-4232-a340-50b47283d82c" (UID: "28c9d555-a3c9-4232-a340-50b47283d82c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:56:12.198342 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.198336 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-dshm" (OuterVolumeSpecName: "dshm") pod "28c9d555-a3c9-4232-a340-50b47283d82c" (UID: "28c9d555-a3c9-4232-a340-50b47283d82c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:56:12.198487 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.198359 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c9d555-a3c9-4232-a340-50b47283d82c-kube-api-access-lr54b" (OuterVolumeSpecName: "kube-api-access-lr54b") pod "28c9d555-a3c9-4232-a340-50b47283d82c" (UID: "28c9d555-a3c9-4232-a340-50b47283d82c"). InnerVolumeSpecName "kube-api-access-lr54b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:56:12.297605 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.297570 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr54b\" (UniqueName: \"kubernetes.io/projected/28c9d555-a3c9-4232-a340-50b47283d82c-kube-api-access-lr54b\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:56:12.297605 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.297599 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c9d555-a3c9-4232-a340-50b47283d82c-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:56:12.297605 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.297610 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c9d555-a3c9-4232-a340-50b47283d82c-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:56:12.931767 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.931735 2552 generic.go:356] "Generic (PLEG): container finished" podID="28c9d555-a3c9-4232-a340-50b47283d82c" containerID="3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc" exitCode=0 Aug 04 00:56:12.932327 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.931777 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" Aug 04 00:56:12.932327 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.931827 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" event={"ID":"28c9d555-a3c9-4232-a340-50b47283d82c","Type":"ContainerDied","Data":"3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc"} Aug 04 00:56:12.932327 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.931892 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2" event={"ID":"28c9d555-a3c9-4232-a340-50b47283d82c","Type":"ContainerDied","Data":"7012cc9886d539ed15b7cbd0ded5b9dfea93ecef881e421f5791e2da42af37b5"} Aug 04 00:56:12.932327 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.931921 2552 scope.go:117] "RemoveContainer" containerID="3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc" Aug 04 00:56:12.942404 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.942377 2552 scope.go:117] "RemoveContainer" containerID="3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc" Aug 04 00:56:12.944324 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:56:12.943922 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc\": container with ID starting with 3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc not found: ID does not exist" containerID="3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc" Aug 04 00:56:12.944324 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.943968 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc"} err="failed to get container status \"3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc\": rpc error: code = NotFound desc = could not find container \"3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc\": container with ID starting with 3b5639f9ffbe709ba11f6752fbd8a2a2a4ea00981e166a9933cab2c69a2413dc not found: ID does not exist" Aug 04 00:56:12.959531 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.959504 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2"] Aug 04 00:56:12.963194 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:12.963173 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/auth-test-model-33bda8-kserve-c67d66987-z8zg2"] Aug 04 00:56:13.953437 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:13.953402 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c9d555-a3c9-4232-a340-50b47283d82c" path="/var/lib/kubelet/pods/28c9d555-a3c9-4232-a340-50b47283d82c/volumes" Aug 04 00:56:17.147221 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.147188 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz"] Aug 04 00:56:17.147585 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.147563 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c9d555-a3c9-4232-a340-50b47283d82c" containerName="main" Aug 04 00:56:17.147585 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.147574 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c9d555-a3c9-4232-a340-50b47283d82c" containerName="main" Aug 04 00:56:17.147668 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.147653 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c9d555-a3c9-4232-a340-50b47283d82c" containerName="main" Aug 04 00:56:17.152040 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.152017 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.155788 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.155659 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5ddf6e\"/\"default-dockercfg-p97hc\"" Aug 04 00:56:17.155930 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.155910 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5ddf6e\"/\"sub-test-model-5ddf6e-kserve-self-signed-certs\"" Aug 04 00:56:17.156163 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.156143 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5ddf6e\"/\"openshift-service-ca.crt\"" Aug 04 00:56:17.156383 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.156366 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5ddf6e\"/\"kube-root-ca.crt\"" Aug 04 00:56:17.163776 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.163753 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz"] Aug 04 00:56:17.241945 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.241913 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/7827a3cc-9619-4480-87db-6b780609a512-kube-api-access-95wq6\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.242133 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.241966 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-model-cache\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.242133 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.241991 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7827a3cc-9619-4480-87db-6b780609a512-tls-certs\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.242133 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.242007 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-tmp-dir\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.242133 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.242032 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-home\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.242133 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.242134 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-dshm\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.342898 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.342838 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/7827a3cc-9619-4480-87db-6b780609a512-kube-api-access-95wq6\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.343088 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.342914 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-model-cache\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.343088 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.342945 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7827a3cc-9619-4480-87db-6b780609a512-tls-certs\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.343088 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.342969 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-tmp-dir\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.343258 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.343097 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-home\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.343258 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.343211 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-dshm\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.343369 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.343335 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-model-cache\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.343428 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.343394 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-tmp-dir\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.343473 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.343454 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-home\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.345294 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.345273 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-dshm\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.345417 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.345401 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7827a3cc-9619-4480-87db-6b780609a512-tls-certs\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.350047 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.350029 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/7827a3cc-9619-4480-87db-6b780609a512-kube-api-access-95wq6\") pod \"sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.463931 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.463824 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:17.598521 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.598495 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz"] Aug 04 00:56:17.600775 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:56:17.600750 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7827a3cc_9619_4480_87db_6b780609a512.slice/crio-42a45905f2e3c537303136d627e68f6ab315e2c5dd4643ba22f756df1561d5c5 WatchSource:0}: Error finding container 42a45905f2e3c537303136d627e68f6ab315e2c5dd4643ba22f756df1561d5c5: Status 404 returned error can't find the container with id 42a45905f2e3c537303136d627e68f6ab315e2c5dd4643ba22f756df1561d5c5 Aug 04 00:56:17.962021 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.961988 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" event={"ID":"7827a3cc-9619-4480-87db-6b780609a512","Type":"ContainerStarted","Data":"b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34"} Aug 04 00:56:17.962021 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.962028 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" event={"ID":"7827a3cc-9619-4480-87db-6b780609a512","Type":"ContainerStarted","Data":"42a45905f2e3c537303136d627e68f6ab315e2c5dd4643ba22f756df1561d5c5"} Aug 04 00:56:17.978226 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:17.978166 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" podStartSLOduration=0.978148284 podStartE2EDuration="978.148284ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:56:17.976164352 +0000 UTC m=+3216.639073842" watchObservedRunningTime="2026-08-04 00:56:17.978148284 +0000 UTC m=+3216.641057762" Aug 04 00:56:27.464371 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:27.464331 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:27.464897 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:27.464395 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:27.477266 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:27.477243 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:56:28.032030 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:56:28.032004 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:57:57.615961 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:57:57.615926 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz"] Aug 04 00:58:12.226924 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.226891 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd"] Aug 04 00:58:12.230329 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.230305 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.233081 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.233051 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5ddf6e\"/\"rate-test-model-5ddf6e-kserve-self-signed-certs\"" Aug 04 00:58:12.242560 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.242540 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd"] Aug 04 00:58:12.299809 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.299783 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mc2p\" (UniqueName: \"kubernetes.io/projected/8be1b352-a4ce-4273-a0b7-34663bf08ddf-kube-api-access-9mc2p\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.299972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.299818 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tls-certs\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.299972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.299842 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-dshm\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.299972 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.299933 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-home\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.300085 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.300040 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tmp-dir\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.300133 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.300089 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-model-cache\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.400690 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.400658 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tmp-dir\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.400899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.400702 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-model-cache\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.400899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.400726 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mc2p\" (UniqueName: \"kubernetes.io/projected/8be1b352-a4ce-4273-a0b7-34663bf08ddf-kube-api-access-9mc2p\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.400899 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.400745 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tls-certs\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.401085 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.400895 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-dshm\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.401085 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.400933 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-home\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.401193 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.401165 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-model-cache\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.401237 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.401190 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tmp-dir\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.401290 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.401271 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-home\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.403085 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.403061 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-dshm\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.403252 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.403234 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tls-certs\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.409336 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.409316 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mc2p\" (UniqueName: \"kubernetes.io/projected/8be1b352-a4ce-4273-a0b7-34663bf08ddf-kube-api-access-9mc2p\") pod \"rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.542329 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.542248 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:12.635040 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.634995 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" podUID="7827a3cc-9619-4480-87db-6b780609a512" containerName="main" containerID="cri-o://b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34" gracePeriod=45 Aug 04 00:58:12.679388 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.679362 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd"] Aug 04 00:58:12.698783 ip-10-0-132-168 kubenswrapper[2552]: W0804 00:58:12.698753 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8be1b352_a4ce_4273_a0b7_34663bf08ddf.slice/crio-6df842ef96751b3478b44b02d091b4422fe5a87e61ca4c9bd196a57c325cf717 WatchSource:0}: Error finding container 6df842ef96751b3478b44b02d091b4422fe5a87e61ca4c9bd196a57c325cf717: Status 404 returned error can't find the container with id 6df842ef96751b3478b44b02d091b4422fe5a87e61ca4c9bd196a57c325cf717 Aug 04 00:58:12.887850 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:12.887827 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:58:13.007509 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.007468 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-model-cache\") pod \"7827a3cc-9619-4480-87db-6b780609a512\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " Aug 04 00:58:13.007711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.007557 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-home\") pod \"7827a3cc-9619-4480-87db-6b780609a512\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " Aug 04 00:58:13.007711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.007591 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-dshm\") pod \"7827a3cc-9619-4480-87db-6b780609a512\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " Aug 04 00:58:13.007711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.007635 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/7827a3cc-9619-4480-87db-6b780609a512-kube-api-access-95wq6\") pod \"7827a3cc-9619-4480-87db-6b780609a512\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " Aug 04 00:58:13.007711 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.007705 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7827a3cc-9619-4480-87db-6b780609a512-tls-certs\") pod \"7827a3cc-9619-4480-87db-6b780609a512\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " Aug 04 00:58:13.007956 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.007754 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-tmp-dir\") pod \"7827a3cc-9619-4480-87db-6b780609a512\" (UID: \"7827a3cc-9619-4480-87db-6b780609a512\") " Aug 04 00:58:13.008513 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.008481 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7827a3cc-9619-4480-87db-6b780609a512" (UID: "7827a3cc-9619-4480-87db-6b780609a512"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:58:13.011261 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.011226 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7827a3cc-9619-4480-87db-6b780609a512-kube-api-access-95wq6" (OuterVolumeSpecName: "kube-api-access-95wq6") pod "7827a3cc-9619-4480-87db-6b780609a512" (UID: "7827a3cc-9619-4480-87db-6b780609a512"). InnerVolumeSpecName "kube-api-access-95wq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:58:13.011594 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.011565 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-dshm" (OuterVolumeSpecName: "dshm") pod "7827a3cc-9619-4480-87db-6b780609a512" (UID: "7827a3cc-9619-4480-87db-6b780609a512"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:58:13.011810 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.011790 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-model-cache" (OuterVolumeSpecName: "model-cache") pod "7827a3cc-9619-4480-87db-6b780609a512" (UID: "7827a3cc-9619-4480-87db-6b780609a512"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:58:13.012001 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.011980 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-home" (OuterVolumeSpecName: "home") pod "7827a3cc-9619-4480-87db-6b780609a512" (UID: "7827a3cc-9619-4480-87db-6b780609a512"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:58:13.014102 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.013957 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7827a3cc-9619-4480-87db-6b780609a512-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7827a3cc-9619-4480-87db-6b780609a512" (UID: "7827a3cc-9619-4480-87db-6b780609a512"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:58:13.109395 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.109319 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/7827a3cc-9619-4480-87db-6b780609a512-kube-api-access-95wq6\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.109395 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.109347 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7827a3cc-9619-4480-87db-6b780609a512-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.109395 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.109357 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.109395 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.109365 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.109395 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.109372 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.109395 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.109379 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7827a3cc-9619-4480-87db-6b780609a512-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.558319 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.558284 2552 generic.go:356] "Generic (PLEG): container finished" podID="7827a3cc-9619-4480-87db-6b780609a512" containerID="b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34" exitCode=0 Aug 04 00:58:13.558319 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.558345 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" Aug 04 00:58:13.558951 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.558369 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" event={"ID":"7827a3cc-9619-4480-87db-6b780609a512","Type":"ContainerDied","Data":"b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34"} Aug 04 00:58:13.558951 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.558416 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz" event={"ID":"7827a3cc-9619-4480-87db-6b780609a512","Type":"ContainerDied","Data":"42a45905f2e3c537303136d627e68f6ab315e2c5dd4643ba22f756df1561d5c5"} Aug 04 00:58:13.558951 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.558438 2552 scope.go:117] "RemoveContainer" containerID="b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34" Aug 04 00:58:13.560223 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.560200 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" event={"ID":"8be1b352-a4ce-4273-a0b7-34663bf08ddf","Type":"ContainerStarted","Data":"763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3"} Aug 04 00:58:13.560336 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.560234 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" event={"ID":"8be1b352-a4ce-4273-a0b7-34663bf08ddf","Type":"ContainerStarted","Data":"6df842ef96751b3478b44b02d091b4422fe5a87e61ca4c9bd196a57c325cf717"} Aug 04 00:58:13.567585 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.567564 2552 scope.go:117] "RemoveContainer" containerID="b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34" Aug 04 00:58:13.567835 ip-10-0-132-168 kubenswrapper[2552]: E0804 00:58:13.567816 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34\": container with ID starting with b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34 not found: ID does not exist" containerID="b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34" Aug 04 00:58:13.567918 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.567842 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34"} err="failed to get container status \"b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34\": rpc error: code = NotFound desc = could not find container \"b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34\": container with ID starting with b8720ba3a71f40a642c854f7d7c1feaedb96d65c99187d36a0fa0c0289287d34 not found: ID does not exist" Aug 04 00:58:13.578409 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.578369 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" podStartSLOduration=1.578354963 podStartE2EDuration="1.578354963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:58:13.576362835 +0000 UTC m=+3332.239272314" watchObservedRunningTime="2026-08-04 00:58:13.578354963 +0000 UTC m=+3332.241264441" Aug 04 00:58:13.590429 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.590405 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz"] Aug 04 00:58:13.593864 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.593833 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/sub-test-model-5ddf6e-kserve-7b6f5976f9-zjnnz"] Aug 04 00:58:13.958846 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:13.958809 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7827a3cc-9619-4480-87db-6b780609a512" path="/var/lib/kubelet/pods/7827a3cc-9619-4480-87db-6b780609a512/volumes" Aug 04 00:58:22.542564 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:22.542530 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:22.542973 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:22.542584 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:22.555398 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:22.555371 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 00:58:22.621628 ip-10-0-132-168 kubenswrapper[2552]: I0804 00:58:22.621599 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 01:00:00.157558 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.157474 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v"] Aug 04 01:00:00.158176 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.158095 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7827a3cc-9619-4480-87db-6b780609a512" containerName="main" Aug 04 01:00:00.158176 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.158117 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7827a3cc-9619-4480-87db-6b780609a512" containerName="main" Aug 04 01:00:00.158298 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.158199 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7827a3cc-9619-4480-87db-6b780609a512" containerName="main" Aug 04 01:00:00.161517 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.161492 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq"] Aug 04 01:00:00.161676 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.161613 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v" Aug 04 01:00:00.164815 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.164793 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 01:00:00.165463 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.165441 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq" Aug 04 01:00:00.173522 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.173497 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v" Aug 04 01:00:00.174216 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.174189 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq"] Aug 04 01:00:00.182049 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.182030 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq" Aug 04 01:00:00.183468 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.183448 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v"] Aug 04 01:00:00.343599 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.343567 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v"] Aug 04 01:00:00.345161 ip-10-0-132-168 kubenswrapper[2552]: W0804 01:00:00.345133 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27425e8b_6585_4d6b_b662_cb6261582fc0.slice/crio-fcb690f9d4cdd3631e00201b8f03614af7505b111144db1781ff61b08d056beb WatchSource:0}: Error finding container fcb690f9d4cdd3631e00201b8f03614af7505b111144db1781ff61b08d056beb: Status 404 returned error can't find the container with id fcb690f9d4cdd3631e00201b8f03614af7505b111144db1781ff61b08d056beb Aug 04 01:00:00.363168 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.363100 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq"] Aug 04 01:00:00.365105 ip-10-0-132-168 kubenswrapper[2552]: W0804 01:00:00.365083 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3e5295_76c7_4c2f_84b7_5b927c2d7f75.slice/crio-4a27e4c7687ac16832b4add8567f639f53e1d53ee109cf786bc058ab0418c8f3 WatchSource:0}: Error finding container 4a27e4c7687ac16832b4add8567f639f53e1d53ee109cf786bc058ab0418c8f3: Status 404 returned error can't find the container with id 4a27e4c7687ac16832b4add8567f639f53e1d53ee109cf786bc058ab0418c8f3 Aug 04 01:00:00.847962 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:00.847877 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd"] Aug 04 01:00:01.101115 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:01.101008 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq" event={"ID":"4b3e5295-76c7-4c2f-84b7-5b927c2d7f75","Type":"ContainerStarted","Data":"a8bc21e7b3b16843a2044da393ba95d50415eb9448768119171d01071261034f"} Aug 04 01:00:01.101115 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:01.101080 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq" event={"ID":"4b3e5295-76c7-4c2f-84b7-5b927c2d7f75","Type":"ContainerStarted","Data":"4a27e4c7687ac16832b4add8567f639f53e1d53ee109cf786bc058ab0418c8f3"} Aug 04 01:00:01.102402 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:01.102368 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v" event={"ID":"27425e8b-6585-4d6b-b662-cb6261582fc0","Type":"ContainerStarted","Data":"34fe7656f7a7870ee4912c516b0bfb2643edde6fcdcc2a3b900bae92548f5c65"} Aug 04 01:00:01.102536 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:01.102412 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v" event={"ID":"27425e8b-6585-4d6b-b662-cb6261582fc0","Type":"ContainerStarted","Data":"fcb690f9d4cdd3631e00201b8f03614af7505b111144db1781ff61b08d056beb"} Aug 04 01:00:01.115787 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:01.115738 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq" podStartSLOduration=1.11572255 podStartE2EDuration="1.11572255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:01.113812282 +0000 UTC m=+3439.776721761" watchObservedRunningTime="2026-08-04 01:00:01.11572255 +0000 UTC m=+3439.778632028" Aug 04 01:00:01.127871 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:01.127819 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v" podStartSLOduration=1.1278046449999999 podStartE2EDuration="1.127804645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:01.126342918 +0000 UTC m=+3439.789252396" watchObservedRunningTime="2026-08-04 01:00:01.127804645 +0000 UTC m=+3439.790714123" Aug 04 01:00:02.109261 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:02.109227 2552 generic.go:356] "Generic (PLEG): container finished" podID="27425e8b-6585-4d6b-b662-cb6261582fc0" containerID="34fe7656f7a7870ee4912c516b0bfb2643edde6fcdcc2a3b900bae92548f5c65" exitCode=0 Aug 04 01:00:02.109692 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:02.109310 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v" event={"ID":"27425e8b-6585-4d6b-b662-cb6261582fc0","Type":"ContainerDied","Data":"34fe7656f7a7870ee4912c516b0bfb2643edde6fcdcc2a3b900bae92548f5c65"} Aug 04 01:00:02.110948 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:02.110926 2552 generic.go:356] "Generic (PLEG): container finished" podID="4b3e5295-76c7-4c2f-84b7-5b927c2d7f75" containerID="a8bc21e7b3b16843a2044da393ba95d50415eb9448768119171d01071261034f" exitCode=0 Aug 04 01:00:02.111077 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:02.110969 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq" event={"ID":"4b3e5295-76c7-4c2f-84b7-5b927c2d7f75","Type":"ContainerDied","Data":"a8bc21e7b3b16843a2044da393ba95d50415eb9448768119171d01071261034f"} Aug 04 01:00:03.274335 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:03.274309 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq" Aug 04 01:00:03.277390 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:03.277368 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v" Aug 04 01:00:04.121631 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:04.121600 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v" Aug 04 01:00:04.121824 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:04.121637 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763420-xmw7v" event={"ID":"27425e8b-6585-4d6b-b662-cb6261582fc0","Type":"ContainerDied","Data":"fcb690f9d4cdd3631e00201b8f03614af7505b111144db1781ff61b08d056beb"} Aug 04 01:00:04.121824 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:04.121682 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcb690f9d4cdd3631e00201b8f03614af7505b111144db1781ff61b08d056beb" Aug 04 01:00:04.123047 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:04.123013 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq" event={"ID":"4b3e5295-76c7-4c2f-84b7-5b927c2d7f75","Type":"ContainerDied","Data":"4a27e4c7687ac16832b4add8567f639f53e1d53ee109cf786bc058ab0418c8f3"} Aug 04 01:00:04.123047 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:04.123027 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq" Aug 04 01:00:04.123252 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:04.123061 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a27e4c7687ac16832b4add8567f639f53e1d53ee109cf786bc058ab0418c8f3" Aug 04 01:00:15.867133 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:15.867072 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" podUID="8be1b352-a4ce-4273-a0b7-34663bf08ddf" containerName="main" containerID="cri-o://763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3" gracePeriod=45 Aug 04 01:00:16.119302 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.119237 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 01:00:16.155455 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155425 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-model-cache\") pod \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " Aug 04 01:00:16.155631 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155485 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tls-certs\") pod \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " Aug 04 01:00:16.155631 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155502 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mc2p\" (UniqueName: \"kubernetes.io/projected/8be1b352-a4ce-4273-a0b7-34663bf08ddf-kube-api-access-9mc2p\") pod \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " Aug 04 01:00:16.155631 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155544 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tmp-dir\") pod \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " Aug 04 01:00:16.155631 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155560 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-dshm\") pod \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " Aug 04 01:00:16.155631 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155586 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-home\") pod \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\" (UID: \"8be1b352-a4ce-4273-a0b7-34663bf08ddf\") " Aug 04 01:00:16.155945 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155720 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-model-cache" (OuterVolumeSpecName: "model-cache") pod "8be1b352-a4ce-4273-a0b7-34663bf08ddf" (UID: "8be1b352-a4ce-4273-a0b7-34663bf08ddf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:00:16.155945 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155836 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8be1b352-a4ce-4273-a0b7-34663bf08ddf" (UID: "8be1b352-a4ce-4273-a0b7-34663bf08ddf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:00:16.155945 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155935 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-home" (OuterVolumeSpecName: "home") pod "8be1b352-a4ce-4273-a0b7-34663bf08ddf" (UID: "8be1b352-a4ce-4273-a0b7-34663bf08ddf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:00:16.156109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.155999 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:00:16.156109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.156034 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:00:16.156109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.156044 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:00:16.157741 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.157712 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-dshm" (OuterVolumeSpecName: "dshm") pod "8be1b352-a4ce-4273-a0b7-34663bf08ddf" (UID: "8be1b352-a4ce-4273-a0b7-34663bf08ddf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:00:16.157913 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.157887 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8be1b352-a4ce-4273-a0b7-34663bf08ddf" (UID: "8be1b352-a4ce-4273-a0b7-34663bf08ddf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 01:00:16.158078 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.158060 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be1b352-a4ce-4273-a0b7-34663bf08ddf-kube-api-access-9mc2p" (OuterVolumeSpecName: "kube-api-access-9mc2p") pod "8be1b352-a4ce-4273-a0b7-34663bf08ddf" (UID: "8be1b352-a4ce-4273-a0b7-34663bf08ddf"). InnerVolumeSpecName "kube-api-access-9mc2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:00:16.187803 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.187763 2552 generic.go:356] "Generic (PLEG): container finished" podID="8be1b352-a4ce-4273-a0b7-34663bf08ddf" containerID="763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3" exitCode=0 Aug 04 01:00:16.188010 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.187809 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" Aug 04 01:00:16.188010 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.187851 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" event={"ID":"8be1b352-a4ce-4273-a0b7-34663bf08ddf","Type":"ContainerDied","Data":"763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3"} Aug 04 01:00:16.188010 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.187929 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd" event={"ID":"8be1b352-a4ce-4273-a0b7-34663bf08ddf","Type":"ContainerDied","Data":"6df842ef96751b3478b44b02d091b4422fe5a87e61ca4c9bd196a57c325cf717"} Aug 04 01:00:16.188010 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.187954 2552 scope.go:117] "RemoveContainer" containerID="763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3" Aug 04 01:00:16.196796 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.196779 2552 scope.go:117] "RemoveContainer" containerID="763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3" Aug 04 01:00:16.197039 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:00:16.197020 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3\": container with ID starting with 763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3 not found: ID does not exist" containerID="763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3" Aug 04 01:00:16.197093 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.197051 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3"} err="failed to get container status \"763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3\": rpc error: code = NotFound desc = could not find container \"763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3\": container with ID starting with 763b701961203b80b894758ae5c839bc2d25dc44a194d996823a21fa9ea4aeb3 not found: ID does not exist" Aug 04 01:00:16.208377 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.208355 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd"] Aug 04 01:00:16.212119 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.212097 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/rate-test-model-5ddf6e-kserve-85f48d65f5-5jwjd"] Aug 04 01:00:16.256827 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.256803 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1b352-a4ce-4273-a0b7-34663bf08ddf-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:00:16.256827 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.256823 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mc2p\" (UniqueName: \"kubernetes.io/projected/8be1b352-a4ce-4273-a0b7-34663bf08ddf-kube-api-access-9mc2p\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:00:16.256993 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:16.256833 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1b352-a4ce-4273-a0b7-34663bf08ddf-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:00:17.953532 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:17.953497 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be1b352-a4ce-4273-a0b7-34663bf08ddf" path="/var/lib/kubelet/pods/8be1b352-a4ce-4273-a0b7-34663bf08ddf/volumes" Aug 04 01:00:18.760796 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.760763 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:18.761235 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.761215 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be1b352-a4ce-4273-a0b7-34663bf08ddf" containerName="main" Aug 04 01:00:18.761235 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.761234 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be1b352-a4ce-4273-a0b7-34663bf08ddf" containerName="main" Aug 04 01:00:18.761379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.761256 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27425e8b-6585-4d6b-b662-cb6261582fc0" containerName="cleanup" Aug 04 01:00:18.761379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.761262 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="27425e8b-6585-4d6b-b662-cb6261582fc0" containerName="cleanup" Aug 04 01:00:18.761379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.761268 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3e5295-76c7-4c2f-84b7-5b927c2d7f75" containerName="cleanup" Aug 04 01:00:18.761379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.761275 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3e5295-76c7-4c2f-84b7-5b927c2d7f75" containerName="cleanup" Aug 04 01:00:18.761379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.761335 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be1b352-a4ce-4273-a0b7-34663bf08ddf" containerName="main" Aug 04 01:00:18.761379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.761341 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b3e5295-76c7-4c2f-84b7-5b927c2d7f75" containerName="cleanup" Aug 04 01:00:18.761379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.761348 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="27425e8b-6585-4d6b-b662-cb6261582fc0" containerName="cleanup" Aug 04 01:00:18.765645 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.765612 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:18.768378 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.768359 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ddmsh\"" Aug 04 01:00:18.772291 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.772267 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:18.879745 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.879712 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqqff\" (UniqueName: \"kubernetes.io/projected/9effa30f-5006-474d-a1a0-194bf55a9ba9-kube-api-access-cqqff\") pod \"test-curl-7541-140011695350096\" (UID: \"9effa30f-5006-474d-a1a0-194bf55a9ba9\") " pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:18.980765 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.980732 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqqff\" (UniqueName: \"kubernetes.io/projected/9effa30f-5006-474d-a1a0-194bf55a9ba9-kube-api-access-cqqff\") pod \"test-curl-7541-140011695350096\" (UID: \"9effa30f-5006-474d-a1a0-194bf55a9ba9\") " pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:18.990738 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:18.990707 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqqff\" (UniqueName: \"kubernetes.io/projected/9effa30f-5006-474d-a1a0-194bf55a9ba9-kube-api-access-cqqff\") pod \"test-curl-7541-140011695350096\" (UID: \"9effa30f-5006-474d-a1a0-194bf55a9ba9\") " pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:19.075851 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:19.075773 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:19.208278 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:19.208252 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:19.210713 ip-10-0-132-168 kubenswrapper[2552]: W0804 01:00:19.210685 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9effa30f_5006_474d_a1a0_194bf55a9ba9.slice/crio-0052e6bcacb417ac451aae83eced56296a8475960ae2c74d8a8883157359ea1b WatchSource:0}: Error finding container 0052e6bcacb417ac451aae83eced56296a8475960ae2c74d8a8883157359ea1b: Status 404 returned error can't find the container with id 0052e6bcacb417ac451aae83eced56296a8475960ae2c74d8a8883157359ea1b Aug 04 01:00:19.212304 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:19.212284 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 01:00:20.210690 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:20.210590 2552 generic.go:356] "Generic (PLEG): container finished" podID="9effa30f-5006-474d-a1a0-194bf55a9ba9" containerID="71cf44fcd468994d874b51d01bb42c7dee447ebc9c367d837f159de2c165e99d" exitCode=0 Aug 04 01:00:20.210690 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:20.210679 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695350096" event={"ID":"9effa30f-5006-474d-a1a0-194bf55a9ba9","Type":"ContainerDied","Data":"71cf44fcd468994d874b51d01bb42c7dee447ebc9c367d837f159de2c165e99d"} Aug 04 01:00:20.211130 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:20.210720 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695350096" event={"ID":"9effa30f-5006-474d-a1a0-194bf55a9ba9","Type":"ContainerStarted","Data":"0052e6bcacb417ac451aae83eced56296a8475960ae2c74d8a8883157359ea1b"} Aug 04 01:00:21.354528 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:21.354503 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:21.391840 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:21.391811 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7541-140011695350096_9effa30f-5006-474d-a1a0-194bf55a9ba9/test-curl-7541-140011695350096/0.log" Aug 04 01:00:21.404059 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:21.404032 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqqff\" (UniqueName: \"kubernetes.io/projected/9effa30f-5006-474d-a1a0-194bf55a9ba9-kube-api-access-cqqff\") pod \"9effa30f-5006-474d-a1a0-194bf55a9ba9\" (UID: \"9effa30f-5006-474d-a1a0-194bf55a9ba9\") " Aug 04 01:00:21.406246 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:21.406223 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9effa30f-5006-474d-a1a0-194bf55a9ba9-kube-api-access-cqqff" (OuterVolumeSpecName: "kube-api-access-cqqff") pod "9effa30f-5006-474d-a1a0-194bf55a9ba9" (UID: "9effa30f-5006-474d-a1a0-194bf55a9ba9"). InnerVolumeSpecName "kube-api-access-cqqff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:00:21.415824 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:21.415803 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:21.417609 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:21.417587 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:21.505235 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:21.505152 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqqff\" (UniqueName: \"kubernetes.io/projected/9effa30f-5006-474d-a1a0-194bf55a9ba9-kube-api-access-cqqff\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:00:21.953436 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:21.953395 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9effa30f-5006-474d-a1a0-194bf55a9ba9" path="/var/lib/kubelet/pods/9effa30f-5006-474d-a1a0-194bf55a9ba9/volumes" Aug 04 01:00:22.163266 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.163236 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r"] Aug 04 01:00:22.163708 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.163685 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9effa30f-5006-474d-a1a0-194bf55a9ba9" containerName="test-curl-7541-140011695350096" Aug 04 01:00:22.163708 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.163704 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9effa30f-5006-474d-a1a0-194bf55a9ba9" containerName="test-curl-7541-140011695350096" Aug 04 01:00:22.163890 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.163785 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9effa30f-5006-474d-a1a0-194bf55a9ba9" containerName="test-curl-7541-140011695350096" Aug 04 01:00:22.167018 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.166996 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.169340 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.169320 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0825a6-openshift-default-dockercfg-t62gq\"" Aug 04 01:00:22.179989 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.179967 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r"] Aug 04 01:00:22.222591 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.222507 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:22.222591 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.222538 2552 scope.go:117] "RemoveContainer" containerID="71cf44fcd468994d874b51d01bb42c7dee447ebc9c367d837f159de2c165e99d" Aug 04 01:00:22.311949 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.311911 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-certs\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.311949 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.311952 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-token\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.312166 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.311981 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-data\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.312166 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.312071 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-podinfo\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.312166 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.312111 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zwh\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-kube-api-access-p8zwh\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.312166 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.312144 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-socket\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.312166 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.312160 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-credential-socket\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.312325 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.312175 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istiod-ca-cert\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.312325 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.312229 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-envoy\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413343 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413306 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-podinfo\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413826 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413355 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zwh\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-kube-api-access-p8zwh\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413826 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413380 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-socket\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413826 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413405 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-credential-socket\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413826 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413429 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istiod-ca-cert\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413826 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413473 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-envoy\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413826 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413521 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-certs\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413826 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413695 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-token\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413826 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413758 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-data\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.413826 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413794 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-credential-socket\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.414392 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.413846 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-socket\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.414392 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.414120 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-certs\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.414392 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.414163 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-data\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.414392 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.414233 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istiod-ca-cert\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.415916 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.415889 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-envoy\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.416118 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.416098 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-podinfo\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.424305 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.424283 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zwh\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-kube-api-access-p8zwh\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.424403 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.424364 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-token\") pod \"e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.479527 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.479451 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:22.620066 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.620042 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r"] Aug 04 01:00:22.622215 ip-10-0-132-168 kubenswrapper[2552]: W0804 01:00:22.622185 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5184eb1_98c7_4bcf_b37e_3dc847c79a37.slice/crio-54c551a0f692810ca3d55df64a0b40ea73c6df0e1451d11961618c3a3dff8a97 WatchSource:0}: Error finding container 54c551a0f692810ca3d55df64a0b40ea73c6df0e1451d11961618c3a3dff8a97: Status 404 returned error can't find the container with id 54c551a0f692810ca3d55df64a0b40ea73c6df0e1451d11961618c3a3dff8a97 Aug 04 01:00:22.624365 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.624337 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:00:22.624457 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.624397 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:00:22.624457 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:22.624427 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:00:23.229841 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:23.229805 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" event={"ID":"d5184eb1-98c7-4bcf-b37e-3dc847c79a37","Type":"ContainerStarted","Data":"edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a"} Aug 04 01:00:23.230010 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:23.229895 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" event={"ID":"d5184eb1-98c7-4bcf-b37e-3dc847c79a37","Type":"ContainerStarted","Data":"54c551a0f692810ca3d55df64a0b40ea73c6df0e1451d11961618c3a3dff8a97"} Aug 04 01:00:23.247907 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:23.247846 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" podStartSLOduration=1.247831342 podStartE2EDuration="1.247831342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:23.245882876 +0000 UTC m=+3461.908792352" watchObservedRunningTime="2026-08-04 01:00:23.247831342 +0000 UTC m=+3461.910740817" Aug 04 01:00:23.479970 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:23.479888 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:23.485179 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:23.485153 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:24.234839 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:24.234728 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:24.235747 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:24.235726 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:28.921532 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:28.921494 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd"] Aug 04 01:00:28.926282 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:28.926254 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:28.932237 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:28.932212 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0825a6\"" Aug 04 01:00:28.934025 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:28.934005 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0825a6-dockercfg-5fvtg\"" Aug 04 01:00:28.940900 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:28.940880 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd"] Aug 04 01:00:29.075253 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:29.075215 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80f7431-03bd-4c63-8d31-ca3b70d68c73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd\" (UID: \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:29.075440 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:29.075264 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2gqv\" (UniqueName: \"kubernetes.io/projected/b80f7431-03bd-4c63-8d31-ca3b70d68c73-kube-api-access-c2gqv\") pod \"payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd\" (UID: \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:29.176417 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:29.176319 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2gqv\" (UniqueName: \"kubernetes.io/projected/b80f7431-03bd-4c63-8d31-ca3b70d68c73-kube-api-access-c2gqv\") pod \"payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd\" (UID: \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:29.176589 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:29.176432 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80f7431-03bd-4c63-8d31-ca3b70d68c73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd\" (UID: \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:29.177127 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:29.177107 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80f7431-03bd-4c63-8d31-ca3b70d68c73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd\" (UID: \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:29.185254 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:29.185227 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2gqv\" (UniqueName: \"kubernetes.io/projected/b80f7431-03bd-4c63-8d31-ca3b70d68c73-kube-api-access-c2gqv\") pod \"payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd\" (UID: \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:29.236840 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:29.236810 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:29.370241 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:29.370216 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd"] Aug 04 01:00:29.372314 ip-10-0-132-168 kubenswrapper[2552]: W0804 01:00:29.372275 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80f7431_03bd_4c63_8d31_ca3b70d68c73.slice/crio-004351cb879b50655463c9915f2350ed59beef7babb25fb0f597a9af02e7efdc WatchSource:0}: Error finding container 004351cb879b50655463c9915f2350ed59beef7babb25fb0f597a9af02e7efdc: Status 404 returned error can't find the container with id 004351cb879b50655463c9915f2350ed59beef7babb25fb0f597a9af02e7efdc Aug 04 01:00:30.266808 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:30.266770 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" event={"ID":"b80f7431-03bd-4c63-8d31-ca3b70d68c73","Type":"ContainerStarted","Data":"94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8"} Aug 04 01:00:30.267273 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:30.266827 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" event={"ID":"b80f7431-03bd-4c63-8d31-ca3b70d68c73","Type":"ContainerStarted","Data":"004351cb879b50655463c9915f2350ed59beef7babb25fb0f597a9af02e7efdc"} Aug 04 01:00:30.267273 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:30.266995 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:30.283344 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:30.283294 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" podStartSLOduration=2.283280565 podStartE2EDuration="2.283280565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:30.281244948 +0000 UTC m=+3468.944154427" watchObservedRunningTime="2026-08-04 01:00:30.283280565 +0000 UTC m=+3468.946190043" Aug 04 01:00:40.278039 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:40.277988 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 01:00:40.278535 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:40.278067 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 01:00:41.275066 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:41.274987 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:00:41.336274 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:41.336195 2552 generic.go:356] "Generic (PLEG): container finished" podID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerID="edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a" exitCode=0 Aug 04 01:00:41.336625 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:41.336268 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" event={"ID":"d5184eb1-98c7-4bcf-b37e-3dc847c79a37","Type":"ContainerDied","Data":"edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a"} Aug 04 01:00:41.336668 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:41.336659 2552 scope.go:117] "RemoveContainer" containerID="edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a" Aug 04 01:00:41.337696 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:41.337673 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:00:41.337757 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:41.337743 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:00:41.337793 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:41.337784 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:00:42.343541 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:42.343514 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" event={"ID":"d5184eb1-98c7-4bcf-b37e-3dc847c79a37","Type":"ContainerStarted","Data":"453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d"} Aug 04 01:00:42.481110 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:42.481073 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:42.481496 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:42.481463 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 01:00:42.481608 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:42.481542 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 01:00:43.479989 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:43.479954 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 01:00:43.480565 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:43.480529 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 01:00:44.481079 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:44.480613 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 01:00:44.481079 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:44.480676 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 01:00:45.480876 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.480815 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 01:00:45.481092 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.480904 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 01:00:45.890815 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.890773 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b"] Aug 04 01:00:45.907954 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.907921 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:45.910639 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.910611 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b"] Aug 04 01:00:45.911572 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.911256 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1cbf8c-openshift-default-dockercfg-jsqw4\"" Aug 04 01:00:45.941680 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.941651 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-podinfo\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:45.941811 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.941693 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ccn\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-kube-api-access-64ccn\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:45.941811 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.941719 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-envoy\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:45.941920 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.941801 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-token\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:45.941920 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.941835 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-socket\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:45.941920 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.941877 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-certs\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:45.942017 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.941920 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccbe229-aa81-42d1-915d-758ad4ca1321-istiod-ca-cert\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:45.942017 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.941938 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-credential-socket\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:45.942017 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:45.941991 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-data\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.043408 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.043374 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ccn\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-kube-api-access-64ccn\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.043408 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.043408 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-envoy\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.043624 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.043538 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-token\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.043624 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.043572 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-socket\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.043624 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.043596 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-certs\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.043726 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.043658 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccbe229-aa81-42d1-915d-758ad4ca1321-istiod-ca-cert\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.043818 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.043788 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-credential-socket\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.043916 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.043898 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-data\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.044037 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.043971 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-socket\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.044037 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.044007 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-podinfo\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.044037 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.044026 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-certs\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.044226 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.044190 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-credential-socket\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.044309 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.044292 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-data\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.044457 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.044436 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccbe229-aa81-42d1-915d-758ad4ca1321-istiod-ca-cert\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.045946 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.045919 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-envoy\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.046285 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.046267 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-podinfo\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.056701 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.056672 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-token\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.058348 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.058327 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ccn\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-kube-api-access-64ccn\") pod \"e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.223373 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.223283 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:46.402781 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.402757 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b"] Aug 04 01:00:46.405246 ip-10-0-132-168 kubenswrapper[2552]: W0804 01:00:46.405216 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccbe229_aa81_42d1_915d_758ad4ca1321.slice/crio-77981d1683194649681389e35e84df65b625d2d5f1f01d2a21c9bb7ec209a575 WatchSource:0}: Error finding container 77981d1683194649681389e35e84df65b625d2d5f1f01d2a21c9bb7ec209a575: Status 404 returned error can't find the container with id 77981d1683194649681389e35e84df65b625d2d5f1f01d2a21c9bb7ec209a575 Aug 04 01:00:46.407588 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.407559 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:00:46.407639 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.407624 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:00:46.407679 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.407654 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:00:46.480907 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.480435 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 01:00:46.480907 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:46.480489 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 01:00:47.371716 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:47.371668 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" event={"ID":"5ccbe229-aa81-42d1-915d-758ad4ca1321","Type":"ContainerStarted","Data":"8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9"} Aug 04 01:00:47.372172 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:47.371749 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" event={"ID":"5ccbe229-aa81-42d1-915d-758ad4ca1321","Type":"ContainerStarted","Data":"77981d1683194649681389e35e84df65b625d2d5f1f01d2a21c9bb7ec209a575"} Aug 04 01:00:47.396698 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:47.396650 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" podStartSLOduration=2.396633784 podStartE2EDuration="2.396633784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:47.394174434 +0000 UTC m=+3486.057083912" watchObservedRunningTime="2026-08-04 01:00:47.396633784 +0000 UTC m=+3486.059543262" Aug 04 01:00:47.483894 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:47.483836 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:47.484270 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:47.484248 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:47.484962 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:47.484943 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:00:48.223943 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:48.223905 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:48.229186 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:48.229159 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:48.376569 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:48.376536 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:00:48.377396 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:00:48.377377 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:01:03.613382 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:03.612439 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": read tcp 10.133.0.2:37692->10.133.0.76:15021: read: connection reset by peer" start-of-body= Aug 04 01:01:03.613382 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:03.612524 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": read tcp 10.133.0.2:37692->10.133.0.76:15021: read: connection reset by peer" Aug 04 01:01:04.463286 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:04.463250 2552 generic.go:356] "Generic (PLEG): container finished" podID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerID="8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9" exitCode=0 Aug 04 01:01:04.463480 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:04.463326 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" event={"ID":"5ccbe229-aa81-42d1-915d-758ad4ca1321","Type":"ContainerDied","Data":"8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9"} Aug 04 01:01:04.463771 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:04.463758 2552 scope.go:117] "RemoveContainer" containerID="8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9" Aug 04 01:01:04.464950 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:04.464926 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:01:04.465018 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:04.464995 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:01:04.465054 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:04.465026 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:01:05.472341 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:05.472297 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" event={"ID":"5ccbe229-aa81-42d1-915d-758ad4ca1321","Type":"ContainerStarted","Data":"e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426"} Aug 04 01:01:06.223842 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:06.223809 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:01:06.224055 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:06.224001 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 01:01:06.224105 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:06.224053 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 01:01:07.224479 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:07.224445 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 01:01:07.224888 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:07.224513 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 01:01:08.224483 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:08.224448 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 01:01:08.224907 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:08.224504 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 01:01:09.223809 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:09.223773 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 01:01:09.224020 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:09.223837 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 01:01:10.224509 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:10.224465 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 01:01:10.224939 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:10.224524 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 01:01:11.227563 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:11.227529 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:01:11.228079 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:11.227841 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:01:11.228504 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:11.228484 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:01:21.067835 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:01:21.067805 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-j2q5h_d1436586-9a7f-4366-a7d1-c504cbac172e/payload-processing/0.log" Aug 04 01:02:11.910693 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:11.910664 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd_b80f7431-03bd-4c63-8d31-ca3b70d68c73/payload-processing/0.log" Aug 04 01:02:12.030404 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.030370 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-j2q5h_d1436586-9a7f-4366-a7d1-c504cbac172e/payload-processing/0.log" Aug 04 01:02:12.679433 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.679399 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg"] Aug 04 01:02:12.683207 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.683183 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.685620 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.685598 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-26288f-openshift-default-dockercfg-xnvsx\"" Aug 04 01:02:12.697847 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.697825 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg"] Aug 04 01:02:12.799712 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.799669 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-certs\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.800003 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.799727 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-token\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.800003 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.799773 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-data\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.800003 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.799811 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-credential-socket\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.800003 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.799884 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-socket\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.800003 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.799957 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63177a1f-0b99-4405-98c8-21a8574a74e6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.800293 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.800077 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-podinfo\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.800293 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.800104 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8vnc\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-kube-api-access-l8vnc\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.800293 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.800148 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-envoy\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901037 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901005 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-certs\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901037 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901041 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-token\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901247 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901064 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-data\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901247 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901115 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-credential-socket\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901247 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901162 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-socket\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901247 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901190 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63177a1f-0b99-4405-98c8-21a8574a74e6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901247 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901233 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-podinfo\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901509 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901258 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8vnc\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-kube-api-access-l8vnc\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901509 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901303 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-envoy\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901509 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901389 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-certs\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901509 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901447 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-data\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901727 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901568 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-socket\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.901792 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.901779 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-credential-socket\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.902070 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.902046 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63177a1f-0b99-4405-98c8-21a8574a74e6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.903530 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.903509 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-envoy\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.903759 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.903732 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-podinfo\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.910026 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.910001 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-token\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.910178 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.910137 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8vnc\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-kube-api-access-l8vnc\") pod \"e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:12.995690 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:12.995608 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:13.150593 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:13.150559 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg"] Aug 04 01:02:13.153408 ip-10-0-132-168 kubenswrapper[2552]: W0804 01:02:13.153377 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63177a1f_0b99_4405_98c8_21a8574a74e6.slice/crio-a4858b5442a4b8c00f1f6f7174eb4f69ee61522d27371ea791db9d4151673380 WatchSource:0}: Error finding container a4858b5442a4b8c00f1f6f7174eb4f69ee61522d27371ea791db9d4151673380: Status 404 returned error can't find the container with id a4858b5442a4b8c00f1f6f7174eb4f69ee61522d27371ea791db9d4151673380 Aug 04 01:02:13.155843 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:13.155803 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:02:13.155943 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:13.155907 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:02:13.155984 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:13.155947 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:02:13.829254 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:13.829216 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" event={"ID":"63177a1f-0b99-4405-98c8-21a8574a74e6","Type":"ContainerStarted","Data":"c446687371fee4615e4fb28fa8ccc382ae6222099fa1fcc4eefc6a6c869d4509"} Aug 04 01:02:13.829434 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:13.829275 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" event={"ID":"63177a1f-0b99-4405-98c8-21a8574a74e6","Type":"ContainerStarted","Data":"a4858b5442a4b8c00f1f6f7174eb4f69ee61522d27371ea791db9d4151673380"} Aug 04 01:02:13.848415 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:13.848361 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" podStartSLOduration=1.848346013 podStartE2EDuration="1.848346013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:02:13.847164412 +0000 UTC m=+3572.510073891" watchObservedRunningTime="2026-08-04 01:02:13.848346013 +0000 UTC m=+3572.511255490" Aug 04 01:02:13.995879 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:13.995828 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:15.001394 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:15.001368 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:15.840227 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:15.840197 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:15.841113 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:15.841094 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:31.882979 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:31.882942 2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 01:02:31.883419 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:31.883001 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 01:02:32.938535 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:32.938502 2552 generic.go:356] "Generic (PLEG): container finished" podID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerID="c446687371fee4615e4fb28fa8ccc382ae6222099fa1fcc4eefc6a6c869d4509" exitCode=0 Aug 04 01:02:32.938995 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:32.938554 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" event={"ID":"63177a1f-0b99-4405-98c8-21a8574a74e6","Type":"ContainerDied","Data":"c446687371fee4615e4fb28fa8ccc382ae6222099fa1fcc4eefc6a6c869d4509"} Aug 04 01:02:32.938995 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:32.938986 2552 scope.go:117] "RemoveContainer" containerID="c446687371fee4615e4fb28fa8ccc382ae6222099fa1fcc4eefc6a6c869d4509" Aug 04 01:02:32.940223 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:32.940199 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:02:32.940307 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:32.940263 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:02:32.940346 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:32.940301 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 01:02:33.946052 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:33.946016 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" event={"ID":"63177a1f-0b99-4405-98c8-21a8574a74e6","Type":"ContainerStarted","Data":"ef61c44b7169ca985252ede6abba138cfad9895589f20a59393d8bc3f7923230"} Aug 04 01:02:33.996398 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:33.996364 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:34.996302 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:34.996263 2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Aug 04 01:02:34.996697 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:34.996325 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Aug 04 01:02:35.996466 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:35.996421 2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Aug 04 01:02:35.996880 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:35.996501 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Aug 04 01:02:36.996168 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:36.996128 2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Aug 04 01:02:36.996342 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:36.996207 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Aug 04 01:02:37.996432 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:37.996401 2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Aug 04 01:02:37.996944 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:37.996473 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Aug 04 01:02:39.000053 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:39.000023 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:39.000419 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:39.000404 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:39.000945 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:39.000921 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:02:55.931185 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:55.931145 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg"] Aug 04 01:02:55.931669 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:02:55.931367 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" containerID="cri-o://ef61c44b7169ca985252ede6abba138cfad9895589f20a59393d8bc3f7923230" gracePeriod=30 Aug 04 01:03:01.097175 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.097142 2552 generic.go:356] "Generic (PLEG): container finished" podID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerID="ef61c44b7169ca985252ede6abba138cfad9895589f20a59393d8bc3f7923230" exitCode=0 Aug 04 01:03:01.097574 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.097214 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" event={"ID":"63177a1f-0b99-4405-98c8-21a8574a74e6","Type":"ContainerDied","Data":"ef61c44b7169ca985252ede6abba138cfad9895589f20a59393d8bc3f7923230"} Aug 04 01:03:01.097574 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.097283 2552 scope.go:117] "RemoveContainer" containerID="c446687371fee4615e4fb28fa8ccc382ae6222099fa1fcc4eefc6a6c869d4509" Aug 04 01:03:01.182597 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.182570 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:03:01.203996 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.203918 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-token\") pod \"63177a1f-0b99-4405-98c8-21a8574a74e6\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " Aug 04 01:03:01.203996 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.203966 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63177a1f-0b99-4405-98c8-21a8574a74e6-istiod-ca-cert\") pod \"63177a1f-0b99-4405-98c8-21a8574a74e6\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " Aug 04 01:03:01.203996 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.203993 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-socket\") pod \"63177a1f-0b99-4405-98c8-21a8574a74e6\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.204038 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-certs\") pod \"63177a1f-0b99-4405-98c8-21a8574a74e6\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.204066 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8vnc\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-kube-api-access-l8vnc\") pod \"63177a1f-0b99-4405-98c8-21a8574a74e6\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.204117 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-envoy\") pod \"63177a1f-0b99-4405-98c8-21a8574a74e6\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.204168 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-data\") pod \"63177a1f-0b99-4405-98c8-21a8574a74e6\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.204202 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-podinfo\") pod \"63177a1f-0b99-4405-98c8-21a8574a74e6\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.204335 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63177a1f-0b99-4405-98c8-21a8574a74e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63177a1f-0b99-4405-98c8-21a8574a74e6" (UID: "63177a1f-0b99-4405-98c8-21a8574a74e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.204526 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63177a1f-0b99-4405-98c8-21a8574a74e6" (UID: "63177a1f-0b99-4405-98c8-21a8574a74e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.204621 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-credential-socket\") pod \"63177a1f-0b99-4405-98c8-21a8574a74e6\" (UID: \"63177a1f-0b99-4405-98c8-21a8574a74e6\") " Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.204849 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63177a1f-0b99-4405-98c8-21a8574a74e6" (UID: "63177a1f-0b99-4405-98c8-21a8574a74e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.205053 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "63177a1f-0b99-4405-98c8-21a8574a74e6" (UID: "63177a1f-0b99-4405-98c8-21a8574a74e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.205062 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63177a1f-0b99-4405-98c8-21a8574a74e6" (UID: "63177a1f-0b99-4405-98c8-21a8574a74e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.205145 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.205167 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63177a1f-0b99-4405-98c8-21a8574a74e6-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.205199 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:01.205393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.205216 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:01.207838 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.207321 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63177a1f-0b99-4405-98c8-21a8574a74e6" (UID: "63177a1f-0b99-4405-98c8-21a8574a74e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 01:03:01.207838 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.207341 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "63177a1f-0b99-4405-98c8-21a8574a74e6" (UID: "63177a1f-0b99-4405-98c8-21a8574a74e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:03:01.207838 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.207436 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63177a1f-0b99-4405-98c8-21a8574a74e6" (UID: "63177a1f-0b99-4405-98c8-21a8574a74e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:03:01.207838 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.207450 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-kube-api-access-l8vnc" (OuterVolumeSpecName: "kube-api-access-l8vnc") pod "63177a1f-0b99-4405-98c8-21a8574a74e6" (UID: "63177a1f-0b99-4405-98c8-21a8574a74e6"). InnerVolumeSpecName "kube-api-access-l8vnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:03:01.306220 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.306185 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:01.306220 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.306213 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:01.306220 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.306224 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8vnc\" (UniqueName: \"kubernetes.io/projected/63177a1f-0b99-4405-98c8-21a8574a74e6-kube-api-access-l8vnc\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:01.306446 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.306237 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:01.306446 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:01.306248 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63177a1f-0b99-4405-98c8-21a8574a74e6-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:02.104420 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:02.104390 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" Aug 04 01:03:02.105122 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:02.104422 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg" event={"ID":"63177a1f-0b99-4405-98c8-21a8574a74e6","Type":"ContainerDied","Data":"a4858b5442a4b8c00f1f6f7174eb4f69ee61522d27371ea791db9d4151673380"} Aug 04 01:03:02.105122 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:02.104487 2552 scope.go:117] "RemoveContainer" containerID="ef61c44b7169ca985252ede6abba138cfad9895589f20a59393d8bc3f7923230" Aug 04 01:03:02.122400 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:02.122369 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg"] Aug 04 01:03:02.125819 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:02.125792 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-26288f-openshift-default-859889c4dd-ns2bg"] Aug 04 01:03:03.952898 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:03.952846 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" path="/var/lib/kubelet/pods/63177a1f-0b99-4405-98c8-21a8574a74e6/volumes" Aug 04 01:03:16.599965 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:16.599925 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd"] Aug 04 01:03:16.600349 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:16.600199 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" podUID="b80f7431-03bd-4c63-8d31-ca3b70d68c73" containerName="payload-processing" containerID="cri-o://94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8" gracePeriod=30 Aug 04 01:03:16.884748 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:16.884726 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:03:16.952372 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:16.952343 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2gqv\" (UniqueName: \"kubernetes.io/projected/b80f7431-03bd-4c63-8d31-ca3b70d68c73-kube-api-access-c2gqv\") pod \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\" (UID: \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\") " Aug 04 01:03:16.952552 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:16.952435 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80f7431-03bd-4c63-8d31-ca3b70d68c73-plugins-config-volume\") pod \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\" (UID: \"b80f7431-03bd-4c63-8d31-ca3b70d68c73\") " Aug 04 01:03:16.952797 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:16.952772 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80f7431-03bd-4c63-8d31-ca3b70d68c73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b80f7431-03bd-4c63-8d31-ca3b70d68c73" (UID: "b80f7431-03bd-4c63-8d31-ca3b70d68c73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:03:16.954375 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:16.954350 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80f7431-03bd-4c63-8d31-ca3b70d68c73-kube-api-access-c2gqv" (OuterVolumeSpecName: "kube-api-access-c2gqv") pod "b80f7431-03bd-4c63-8d31-ca3b70d68c73" (UID: "b80f7431-03bd-4c63-8d31-ca3b70d68c73"). InnerVolumeSpecName "kube-api-access-c2gqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:03:17.053486 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.053453 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80f7431-03bd-4c63-8d31-ca3b70d68c73-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:17.053486 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.053482 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2gqv\" (UniqueName: \"kubernetes.io/projected/b80f7431-03bd-4c63-8d31-ca3b70d68c73-kube-api-access-c2gqv\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:03:17.186209 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.186131 2552 generic.go:356] "Generic (PLEG): container finished" podID="b80f7431-03bd-4c63-8d31-ca3b70d68c73" containerID="94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8" exitCode=0 Aug 04 01:03:17.186209 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.186181 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" Aug 04 01:03:17.186393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.186219 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" event={"ID":"b80f7431-03bd-4c63-8d31-ca3b70d68c73","Type":"ContainerDied","Data":"94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8"} Aug 04 01:03:17.186393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.186283 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd" event={"ID":"b80f7431-03bd-4c63-8d31-ca3b70d68c73","Type":"ContainerDied","Data":"004351cb879b50655463c9915f2350ed59beef7babb25fb0f597a9af02e7efdc"} Aug 04 01:03:17.186393 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.186306 2552 scope.go:117] "RemoveContainer" containerID="94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8" Aug 04 01:03:17.195908 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.195891 2552 scope.go:117] "RemoveContainer" containerID="94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8" Aug 04 01:03:17.196149 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:03:17.196132 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8\": container with ID starting with 94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8 not found: ID does not exist" containerID="94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8" Aug 04 01:03:17.196229 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.196158 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8"} err="failed to get container status \"94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8\": rpc error: code = NotFound desc = could not find container \"94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8\": container with ID starting with 94a71fe111cc6d1c7d7d2c375268a012c5d5dc302a6985b25aa2ce1b404091f8 not found: ID does not exist" Aug 04 01:03:17.208332 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.208313 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd"] Aug 04 01:03:17.211616 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.211593 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0825a6-c59b6c7d-9rzhd"] Aug 04 01:03:17.952869 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:03:17.952812 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b80f7431-03bd-4c63-8d31-ca3b70d68c73" path="/var/lib/kubelet/pods/b80f7431-03bd-4c63-8d31-ca3b70d68c73/volumes" Aug 04 01:04:20.675323 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:20.675286 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r"] Aug 04 01:04:20.676186 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:20.675542 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" containerID="cri-o://453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d" gracePeriod=30 Aug 04 01:04:25.930339 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:25.930315 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:04:26.011160 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011078 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-credential-socket\") pod \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " Aug 04 01:04:26.011160 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011121 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8zwh\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-kube-api-access-p8zwh\") pod \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " Aug 04 01:04:26.011379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011179 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istiod-ca-cert\") pod \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " Aug 04 01:04:26.011379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011209 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-token\") pod \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " Aug 04 01:04:26.011379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011281 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-data\") pod \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " Aug 04 01:04:26.011576 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011314 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-socket\") pod \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " Aug 04 01:04:26.011730 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011618 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-certs\") pod \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " Aug 04 01:04:26.011730 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011346 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5184eb1-98c7-4bcf-b37e-3dc847c79a37" (UID: "d5184eb1-98c7-4bcf-b37e-3dc847c79a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:26.011730 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011569 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5184eb1-98c7-4bcf-b37e-3dc847c79a37" (UID: "d5184eb1-98c7-4bcf-b37e-3dc847c79a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:26.011730 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011669 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-podinfo\") pod \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " Aug 04 01:04:26.011730 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011696 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-envoy\") pod \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\" (UID: \"d5184eb1-98c7-4bcf-b37e-3dc847c79a37\") " Aug 04 01:04:26.011730 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011586 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5184eb1-98c7-4bcf-b37e-3dc847c79a37" (UID: "d5184eb1-98c7-4bcf-b37e-3dc847c79a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:26.012109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011630 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5184eb1-98c7-4bcf-b37e-3dc847c79a37" (UID: "d5184eb1-98c7-4bcf-b37e-3dc847c79a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:26.012109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.011846 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5184eb1-98c7-4bcf-b37e-3dc847c79a37" (UID: "d5184eb1-98c7-4bcf-b37e-3dc847c79a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:26.012109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.012047 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:26.012109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.012064 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:26.012109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.012076 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:26.012109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.012088 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:26.012109 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.012101 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:26.013403 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.013381 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5184eb1-98c7-4bcf-b37e-3dc847c79a37" (UID: "d5184eb1-98c7-4bcf-b37e-3dc847c79a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:26.013588 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.013564 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-kube-api-access-p8zwh" (OuterVolumeSpecName: "kube-api-access-p8zwh") pod "d5184eb1-98c7-4bcf-b37e-3dc847c79a37" (UID: "d5184eb1-98c7-4bcf-b37e-3dc847c79a37"). InnerVolumeSpecName "kube-api-access-p8zwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:26.013681 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.013660 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5184eb1-98c7-4bcf-b37e-3dc847c79a37" (UID: "d5184eb1-98c7-4bcf-b37e-3dc847c79a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:26.013726 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.013681 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5184eb1-98c7-4bcf-b37e-3dc847c79a37" (UID: "d5184eb1-98c7-4bcf-b37e-3dc847c79a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 01:04:26.113213 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.113174 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:26.113213 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.113206 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:26.113213 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.113215 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8zwh\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-kube-api-access-p8zwh\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:26.113449 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.113224 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5184eb1-98c7-4bcf-b37e-3dc847c79a37-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:26.539378 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.539345 2552 generic.go:356] "Generic (PLEG): container finished" podID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerID="453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d" exitCode=0 Aug 04 01:04:26.539558 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.539397 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" Aug 04 01:04:26.539558 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.539425 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" event={"ID":"d5184eb1-98c7-4bcf-b37e-3dc847c79a37","Type":"ContainerDied","Data":"453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d"} Aug 04 01:04:26.539558 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.539484 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r" event={"ID":"d5184eb1-98c7-4bcf-b37e-3dc847c79a37","Type":"ContainerDied","Data":"54c551a0f692810ca3d55df64a0b40ea73c6df0e1451d11961618c3a3dff8a97"} Aug 04 01:04:26.539558 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.539507 2552 scope.go:117] "RemoveContainer" containerID="453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d" Aug 04 01:04:26.548508 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.548488 2552 scope.go:117] "RemoveContainer" containerID="edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a" Aug 04 01:04:26.556437 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.556415 2552 scope.go:117] "RemoveContainer" containerID="453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d" Aug 04 01:04:26.556688 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:04:26.556671 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d\": container with ID starting with 453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d not found: ID does not exist" containerID="453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d" Aug 04 01:04:26.556747 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.556695 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d"} err="failed to get container status \"453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d\": rpc error: code = NotFound desc = could not find container \"453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d\": container with ID starting with 453ef93f1ec182c9d3b4fa7d5a16debc2c526e41e4a8cb1326258f806401012d not found: ID does not exist" Aug 04 01:04:26.556747 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.556710 2552 scope.go:117] "RemoveContainer" containerID="edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a" Aug 04 01:04:26.556937 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:04:26.556921 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a\": container with ID starting with edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a not found: ID does not exist" containerID="edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a" Aug 04 01:04:26.556975 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.556940 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a"} err="failed to get container status \"edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a\": rpc error: code = NotFound desc = could not find container \"edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a\": container with ID starting with edf2849d95ea53b80b6d863af7f748a68de2774fb20d407cb81d4144947a9a8a not found: ID does not exist" Aug 04 01:04:26.560566 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.560541 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r"] Aug 04 01:04:26.562529 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:26.562511 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0825a6-openshift-default-54fbffcf4b-77t5r"] Aug 04 01:04:27.952967 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:27.952935 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" path="/var/lib/kubelet/pods/d5184eb1-98c7-4bcf-b37e-3dc847c79a37/volumes" Aug 04 01:04:35.067878 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:35.067638 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b"] Aug 04 01:04:35.071220 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:35.071179 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" containerID="cri-o://e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426" gracePeriod=30 Aug 04 01:04:40.324111 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.324086 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:04:40.446513 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446479 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccbe229-aa81-42d1-915d-758ad4ca1321-istiod-ca-cert\") pod \"5ccbe229-aa81-42d1-915d-758ad4ca1321\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " Aug 04 01:04:40.446693 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446528 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64ccn\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-kube-api-access-64ccn\") pod \"5ccbe229-aa81-42d1-915d-758ad4ca1321\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " Aug 04 01:04:40.446693 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446550 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-token\") pod \"5ccbe229-aa81-42d1-915d-758ad4ca1321\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " Aug 04 01:04:40.446693 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446569 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-data\") pod \"5ccbe229-aa81-42d1-915d-758ad4ca1321\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " Aug 04 01:04:40.446693 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446618 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-podinfo\") pod \"5ccbe229-aa81-42d1-915d-758ad4ca1321\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " Aug 04 01:04:40.446693 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446645 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-envoy\") pod \"5ccbe229-aa81-42d1-915d-758ad4ca1321\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " Aug 04 01:04:40.446693 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446679 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-socket\") pod \"5ccbe229-aa81-42d1-915d-758ad4ca1321\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " Aug 04 01:04:40.447048 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446736 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-certs\") pod \"5ccbe229-aa81-42d1-915d-758ad4ca1321\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " Aug 04 01:04:40.447048 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446810 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-credential-socket\") pod \"5ccbe229-aa81-42d1-915d-758ad4ca1321\" (UID: \"5ccbe229-aa81-42d1-915d-758ad4ca1321\") " Aug 04 01:04:40.447048 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.446908 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccbe229-aa81-42d1-915d-758ad4ca1321-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ccbe229-aa81-42d1-915d-758ad4ca1321" (UID: "5ccbe229-aa81-42d1-915d-758ad4ca1321"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:40.447048 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.447010 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ccbe229-aa81-42d1-915d-758ad4ca1321" (UID: "5ccbe229-aa81-42d1-915d-758ad4ca1321"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:40.447276 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.447101 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ccbe229-aa81-42d1-915d-758ad4ca1321" (UID: "5ccbe229-aa81-42d1-915d-758ad4ca1321"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:40.447276 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.447222 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ccbe229-aa81-42d1-915d-758ad4ca1321" (UID: "5ccbe229-aa81-42d1-915d-758ad4ca1321"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:40.447276 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.447246 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ccbe229-aa81-42d1-915d-758ad4ca1321-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:40.447276 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.447267 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:40.447458 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.447281 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:40.447458 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.447291 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ccbe229-aa81-42d1-915d-758ad4ca1321" (UID: "5ccbe229-aa81-42d1-915d-758ad4ca1321"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:40.448870 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.448837 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ccbe229-aa81-42d1-915d-758ad4ca1321" (UID: "5ccbe229-aa81-42d1-915d-758ad4ca1321"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:40.448967 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.448832 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ccbe229-aa81-42d1-915d-758ad4ca1321" (UID: "5ccbe229-aa81-42d1-915d-758ad4ca1321"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 01:04:40.449143 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.449122 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-kube-api-access-64ccn" (OuterVolumeSpecName: "kube-api-access-64ccn") pod "5ccbe229-aa81-42d1-915d-758ad4ca1321" (UID: "5ccbe229-aa81-42d1-915d-758ad4ca1321"). InnerVolumeSpecName "kube-api-access-64ccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:40.449199 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.449127 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ccbe229-aa81-42d1-915d-758ad4ca1321" (UID: "5ccbe229-aa81-42d1-915d-758ad4ca1321"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:40.547730 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.547698 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ccn\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-kube-api-access-64ccn\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:40.547730 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.547726 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:40.547730 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.547735 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:40.547987 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.547744 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:40.547987 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.547753 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:40.547987 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.547763 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ccbe229-aa81-42d1-915d-758ad4ca1321-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:40.617416 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.617387 2552 generic.go:356] "Generic (PLEG): container finished" podID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerID="e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426" exitCode=0 Aug 04 01:04:40.617565 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.617438 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" Aug 04 01:04:40.617565 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.617472 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" event={"ID":"5ccbe229-aa81-42d1-915d-758ad4ca1321","Type":"ContainerDied","Data":"e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426"} Aug 04 01:04:40.617565 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.617531 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b" event={"ID":"5ccbe229-aa81-42d1-915d-758ad4ca1321","Type":"ContainerDied","Data":"77981d1683194649681389e35e84df65b625d2d5f1f01d2a21c9bb7ec209a575"} Aug 04 01:04:40.617565 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.617561 2552 scope.go:117] "RemoveContainer" containerID="e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426" Aug 04 01:04:40.626901 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.626880 2552 scope.go:117] "RemoveContainer" containerID="8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9" Aug 04 01:04:40.634880 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.634852 2552 scope.go:117] "RemoveContainer" containerID="e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426" Aug 04 01:04:40.635132 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:04:40.635113 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426\": container with ID starting with e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426 not found: ID does not exist" containerID="e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426" Aug 04 01:04:40.635181 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.635140 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426"} err="failed to get container status \"e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426\": rpc error: code = NotFound desc = could not find container \"e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426\": container with ID starting with e83ae9a0e5baa9acc1616386523dc59de554244bfcca134edf60b43af3fba426 not found: ID does not exist" Aug 04 01:04:40.635181 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.635158 2552 scope.go:117] "RemoveContainer" containerID="8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9" Aug 04 01:04:40.635397 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:04:40.635379 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9\": container with ID starting with 8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9 not found: ID does not exist" containerID="8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9" Aug 04 01:04:40.635451 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.635402 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9"} err="failed to get container status \"8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9\": rpc error: code = NotFound desc = could not find container \"8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9\": container with ID starting with 8a31891f6e8f6e42205a117409f94c95e48392c6c5be620e26c51a0177263ad9 not found: ID does not exist" Aug 04 01:04:40.644872 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.644830 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b"] Aug 04 01:04:40.646056 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:40.646034 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1cbf8c-openshift-default-8f4d767db-rbp9b"] Aug 04 01:04:41.953218 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:41.953186 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" path="/var/lib/kubelet/pods/5ccbe229-aa81-42d1-915d-758ad4ca1321/volumes" Aug 04 01:04:47.976646 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:47.976610 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm"] Aug 04 01:04:47.977438 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:47.976844 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" containerID="cri-o://49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29" gracePeriod=30 Aug 04 01:04:52.842529 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:52.842498 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq"] Aug 04 01:04:52.844272 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:52.844252 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-33bda8-29763420-clcpq"] Aug 04 01:04:53.240450 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.240420 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 01:04:53.358313 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358222 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-envoy\") pod \"eed002a2-9025-4a51-9cbb-6919bcb967a3\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " Aug 04 01:04:53.358477 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358319 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-podinfo\") pod \"eed002a2-9025-4a51-9cbb-6919bcb967a3\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " Aug 04 01:04:53.358477 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358350 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed002a2-9025-4a51-9cbb-6919bcb967a3-istiod-ca-cert\") pod \"eed002a2-9025-4a51-9cbb-6919bcb967a3\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " Aug 04 01:04:53.358477 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358366 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-socket\") pod \"eed002a2-9025-4a51-9cbb-6919bcb967a3\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " Aug 04 01:04:53.358477 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358394 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-certs\") pod \"eed002a2-9025-4a51-9cbb-6919bcb967a3\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " Aug 04 01:04:53.358477 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358420 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-data\") pod \"eed002a2-9025-4a51-9cbb-6919bcb967a3\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " Aug 04 01:04:53.358764 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358480 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jff7\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-kube-api-access-7jff7\") pod \"eed002a2-9025-4a51-9cbb-6919bcb967a3\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " Aug 04 01:04:53.358764 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358529 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-token\") pod \"eed002a2-9025-4a51-9cbb-6919bcb967a3\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " Aug 04 01:04:53.358764 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358583 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-credential-socket\") pod \"eed002a2-9025-4a51-9cbb-6919bcb967a3\" (UID: \"eed002a2-9025-4a51-9cbb-6919bcb967a3\") " Aug 04 01:04:53.358764 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358728 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eed002a2-9025-4a51-9cbb-6919bcb967a3" (UID: "eed002a2-9025-4a51-9cbb-6919bcb967a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:53.358764 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358740 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "eed002a2-9025-4a51-9cbb-6919bcb967a3" (UID: "eed002a2-9025-4a51-9cbb-6919bcb967a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:53.358764 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358733 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eed002a2-9025-4a51-9cbb-6919bcb967a3" (UID: "eed002a2-9025-4a51-9cbb-6919bcb967a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:53.359115 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358801 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed002a2-9025-4a51-9cbb-6919bcb967a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eed002a2-9025-4a51-9cbb-6919bcb967a3" (UID: "eed002a2-9025-4a51-9cbb-6919bcb967a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:53.359115 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.358979 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eed002a2-9025-4a51-9cbb-6919bcb967a3" (UID: "eed002a2-9025-4a51-9cbb-6919bcb967a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:53.359115 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.359087 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.359115 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.359106 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed002a2-9025-4a51-9cbb-6919bcb967a3-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.359340 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.359121 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.359340 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.359135 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.359340 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.359163 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.360884 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.360674 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eed002a2-9025-4a51-9cbb-6919bcb967a3" (UID: "eed002a2-9025-4a51-9cbb-6919bcb967a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:04:53.360884 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.360740 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eed002a2-9025-4a51-9cbb-6919bcb967a3" (UID: "eed002a2-9025-4a51-9cbb-6919bcb967a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 01:04:53.360884 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.360764 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "eed002a2-9025-4a51-9cbb-6919bcb967a3" (UID: "eed002a2-9025-4a51-9cbb-6919bcb967a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:53.360884 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.360847 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-kube-api-access-7jff7" (OuterVolumeSpecName: "kube-api-access-7jff7") pod "eed002a2-9025-4a51-9cbb-6919bcb967a3" (UID: "eed002a2-9025-4a51-9cbb-6919bcb967a3"). InnerVolumeSpecName "kube-api-access-7jff7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:53.460506 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.460475 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.460506 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.460501 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.460506 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.460512 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jff7\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-kube-api-access-7jff7\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.460726 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.460522 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed002a2-9025-4a51-9cbb-6919bcb967a3-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.684840 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.684806 2552 generic.go:356] "Generic (PLEG): container finished" podID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerID="49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29" exitCode=0 Aug 04 01:04:53.685008 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.684881 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" Aug 04 01:04:53.685008 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.684899 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" event={"ID":"eed002a2-9025-4a51-9cbb-6919bcb967a3","Type":"ContainerDied","Data":"49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29"} Aug 04 01:04:53.685008 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.684985 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm" event={"ID":"eed002a2-9025-4a51-9cbb-6919bcb967a3","Type":"ContainerDied","Data":"4f2098ad34565274e799e9195b24c690a88ba90f21663b0b0d6a9a7b34f446e0"} Aug 04 01:04:53.685118 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.685009 2552 scope.go:117] "RemoveContainer" containerID="49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29" Aug 04 01:04:53.694214 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.694195 2552 scope.go:117] "RemoveContainer" containerID="10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd" Aug 04 01:04:53.702183 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.702168 2552 scope.go:117] "RemoveContainer" containerID="49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29" Aug 04 01:04:53.702446 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:04:53.702424 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29\": container with ID starting with 49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29 not found: ID does not exist" containerID="49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29" Aug 04 01:04:53.702530 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.702458 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29"} err="failed to get container status \"49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29\": rpc error: code = NotFound desc = could not find container \"49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29\": container with ID starting with 49e63c32c8a751579458f9a707e67e58a7cb5e886a3e9c76b64fac2ba7d84d29 not found: ID does not exist" Aug 04 01:04:53.702530 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.702482 2552 scope.go:117] "RemoveContainer" containerID="10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd" Aug 04 01:04:53.702731 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:04:53.702715 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd\": container with ID starting with 10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd not found: ID does not exist" containerID="10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd" Aug 04 01:04:53.702776 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.702737 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd"} err="failed to get container status \"10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd\": rpc error: code = NotFound desc = could not find container \"10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd\": container with ID starting with 10470c47cbaf4745abc6165240903d90f05cb28a47c9494dbb977333f340e4bd not found: ID does not exist" Aug 04 01:04:53.709926 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.709906 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm"] Aug 04 01:04:53.714134 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.714116 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-5ddf6e-openshift-default-d4ff8f876-24bjm"] Aug 04 01:04:53.953341 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.953266 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b3e5295-76c7-4c2f-84b7-5b927c2d7f75" path="/var/lib/kubelet/pods/4b3e5295-76c7-4c2f-84b7-5b927c2d7f75/volumes" Aug 04 01:04:53.953681 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:53.953661 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" path="/var/lib/kubelet/pods/eed002a2-9025-4a51-9cbb-6919bcb967a3/volumes" Aug 04 01:04:59.700469 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:59.700431 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj"] Aug 04 01:04:59.700899 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:04:59.700694 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" containerID="cri-o://8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda" gracePeriod=30 Aug 04 01:05:00.719609 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:00.719568 2552 patch_prober.go:28] interesting pod/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 01:05:00.719999 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:00.719636 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 01:05:04.954499 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:04.954471 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 01:05:05.064878 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.064749 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-socket\") pod \"6b9dac14-724b-473c-91fd-c294923dd847\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " Aug 04 01:05:05.064878 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.064832 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-data\") pod \"6b9dac14-724b-473c-91fd-c294923dd847\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " Aug 04 01:05:05.064878 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.064885 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9dac14-724b-473c-91fd-c294923dd847-istio-podinfo\") pod \"6b9dac14-724b-473c-91fd-c294923dd847\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " Aug 04 01:05:05.065177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.064902 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-istio-token\") pod \"6b9dac14-724b-473c-91fd-c294923dd847\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " Aug 04 01:05:05.065177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.064935 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9dac14-724b-473c-91fd-c294923dd847-istiod-ca-cert\") pod \"6b9dac14-724b-473c-91fd-c294923dd847\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " Aug 04 01:05:05.065177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.064967 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-envoy\") pod \"6b9dac14-724b-473c-91fd-c294923dd847\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " Aug 04 01:05:05.065177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.064989 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-certs\") pod \"6b9dac14-724b-473c-91fd-c294923dd847\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " Aug 04 01:05:05.065177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.065007 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-credential-socket\") pod \"6b9dac14-724b-473c-91fd-c294923dd847\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " Aug 04 01:05:05.065177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.065053 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95tbc\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-kube-api-access-95tbc\") pod \"6b9dac14-724b-473c-91fd-c294923dd847\" (UID: \"6b9dac14-724b-473c-91fd-c294923dd847\") " Aug 04 01:05:05.065177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.065079 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b9dac14-724b-473c-91fd-c294923dd847" (UID: "6b9dac14-724b-473c-91fd-c294923dd847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:05:05.065177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.065116 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b9dac14-724b-473c-91fd-c294923dd847" (UID: "6b9dac14-724b-473c-91fd-c294923dd847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:05:05.065606 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.065348 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b9dac14-724b-473c-91fd-c294923dd847" (UID: "6b9dac14-724b-473c-91fd-c294923dd847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:05:05.065606 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.065365 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b9dac14-724b-473c-91fd-c294923dd847" (UID: "6b9dac14-724b-473c-91fd-c294923dd847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:05:05.065606 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.065369 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:05:05.065606 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.065413 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:05:05.065800 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.065660 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9dac14-724b-473c-91fd-c294923dd847-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b9dac14-724b-473c-91fd-c294923dd847" (UID: "6b9dac14-724b-473c-91fd-c294923dd847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:05:05.067204 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.067172 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b9dac14-724b-473c-91fd-c294923dd847-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b9dac14-724b-473c-91fd-c294923dd847" (UID: "6b9dac14-724b-473c-91fd-c294923dd847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 01:05:05.067204 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.067190 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-kube-api-access-95tbc" (OuterVolumeSpecName: "kube-api-access-95tbc") pod "6b9dac14-724b-473c-91fd-c294923dd847" (UID: "6b9dac14-724b-473c-91fd-c294923dd847"). InnerVolumeSpecName "kube-api-access-95tbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:05:05.067490 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.067471 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b9dac14-724b-473c-91fd-c294923dd847" (UID: "6b9dac14-724b-473c-91fd-c294923dd847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:05:05.067534 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.067483 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b9dac14-724b-473c-91fd-c294923dd847" (UID: "6b9dac14-724b-473c-91fd-c294923dd847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:05:05.166745 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.166706 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:05:05.166745 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.166738 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95tbc\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-kube-api-access-95tbc\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:05:05.166745 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.166751 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9dac14-724b-473c-91fd-c294923dd847-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:05:05.167016 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.166764 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9dac14-724b-473c-91fd-c294923dd847-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:05:05.167016 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.166776 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9dac14-724b-473c-91fd-c294923dd847-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:05:05.167016 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.166786 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:05:05.167016 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.166796 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9dac14-724b-473c-91fd-c294923dd847-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Aug 04 01:05:05.750737 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.750703 2552 generic.go:356] "Generic (PLEG): container finished" podID="6b9dac14-724b-473c-91fd-c294923dd847" containerID="8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda" exitCode=0 Aug 04 01:05:05.750962 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.750757 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" Aug 04 01:05:05.750962 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.750788 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" event={"ID":"6b9dac14-724b-473c-91fd-c294923dd847","Type":"ContainerDied","Data":"8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda"} Aug 04 01:05:05.750962 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.750882 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj" event={"ID":"6b9dac14-724b-473c-91fd-c294923dd847","Type":"ContainerDied","Data":"54f9b895bf3924be95f39a7bdc310e15dd9afb304418bc95c38334f64e85fabf"} Aug 04 01:05:05.750962 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.750904 2552 scope.go:117] "RemoveContainer" containerID="8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda" Aug 04 01:05:05.759903 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.759879 2552 scope.go:117] "RemoveContainer" containerID="e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d" Aug 04 01:05:05.767380 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.767362 2552 scope.go:117] "RemoveContainer" containerID="8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda" Aug 04 01:05:05.767643 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:05:05.767619 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda\": container with ID starting with 8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda not found: ID does not exist" containerID="8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda" Aug 04 01:05:05.767712 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.767649 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda"} err="failed to get container status \"8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda\": rpc error: code = NotFound desc = could not find container \"8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda\": container with ID starting with 8849fee7c09ed73d9709161fea8afc30e3a48e1d2d2fa0d20bc76e1c9aadffda not found: ID does not exist" Aug 04 01:05:05.767712 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.767666 2552 scope.go:117] "RemoveContainer" containerID="e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d" Aug 04 01:05:05.767925 ip-10-0-132-168 kubenswrapper[2552]: E0804 01:05:05.767907 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d\": container with ID starting with e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d not found: ID does not exist" containerID="e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d" Aug 04 01:05:05.768004 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.767930 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d"} err="failed to get container status \"e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d\": rpc error: code = NotFound desc = could not find container \"e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d\": container with ID starting with e1d6a01212c06349f3cd240c24bd132dfadee9e8706b8c7bf6f7ba33ef12c40d not found: ID does not exist" Aug 04 01:05:05.771496 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.771476 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj"] Aug 04 01:05:05.773142 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.773125 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-33bda8-openshift-default-799969dfbd-p27fj"] Aug 04 01:05:05.952576 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.952545 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b9dac14-724b-473c-91fd-c294923dd847" path="/var/lib/kubelet/pods/6b9dac14-724b-473c-91fd-c294923dd847/volumes" Aug 04 01:05:05.963341 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:05.963315 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-h7qtw_8e648d96-5199-4788-ab69-79b1e3cad502/manager/0.log" Aug 04 01:05:06.088358 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:06.088283 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lnpqr_fe9f0525-cf27-4a1d-9722-698306049b57/manager/0.log" Aug 04 01:05:06.362177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:06.362155 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ddshn_bbce56ae-aa73-41f1-9e9e-08a3573fc8bd/server/0.log" Aug 04 01:05:06.485729 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:06.485686 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gxlrp_08461dce-ee65-4059-9985-ca2af7252bcb/manager/0.log" Aug 04 01:05:06.747323 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:06.747246 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-wgcxn_9efde704-d281-4e4a-8164-160af9c7033a/copy-manifests/0.log" Aug 04 01:05:06.755736 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:06.755715 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-wgcxn_9efde704-d281-4e4a-8164-160af9c7033a/manager/0.log" Aug 04 01:05:07.385177 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:07.385147 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29763390-fbbxd_31ca444d-9971-463c-8dab-b3332d6d5473/cleanup/0.log" Aug 04 01:05:07.615176 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:07.615148 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29763420-xmw7v_27425e8b-6585-4d6b-b662-cb6261582fc0/cleanup/0.log" Aug 04 01:05:08.711359 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:08.711327 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-v2sg7_e5d1cd67-c23b-4afc-829c-42b95129c58d/manager/0.log" Aug 04 01:05:08.840502 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:08.840475 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-fwkml_8c04aaf8-469f-4662-ab08-d7c7f59fa482/manager/0.log" Aug 04 01:05:09.097829 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:09.097731 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-bs9h7_b9e19202-0064-4b6a-9470-eeeb6094955e/manager/0.log" Aug 04 01:05:09.336553 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:09.336512 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-wdpgm_63cfe342-44f8-4c23-8ad7-b1aa22b1a943/manager/0.log" Aug 04 01:05:09.729528 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:09.729500 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q_26863ba2-e38c-48b9-ae4e-95ec614716ff/istio-proxy/0.log" Aug 04 01:05:09.975029 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:09.975004 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-554b4579cb-f2ksc_f20a59f4-5d4e-4796-b7d8-60b9587e8b3d/kube-auth-proxy/0.log" Aug 04 01:05:10.212672 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:10.212643 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8cwk8_9c4e0926-157d-4f36-874d-bf2af64dbc79/istio-proxy/1.log" Aug 04 01:05:10.453550 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:10.453461 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-j2q5h_d1436586-9a7f-4366-a7d1-c504cbac172e/payload-processing/0.log" Aug 04 01:05:11.438379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:11.438353 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-qpcg2_52f4feea-d897-47ae-81d1-bec12aae84d2/main/0.log" Aug 04 01:05:11.561446 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:11.561417 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-kvrbx_1beb34df-a3db-40f6-b76a-1e4d29793d0b/main/0.log" Aug 04 01:05:17.094744 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.094707 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9cdcm/must-gather-cllcw"] Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095062 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095082 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095093 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095099 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095107 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095113 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095121 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095126 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095133 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095138 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095147 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b80f7431-03bd-4c63-8d31-ca3b70d68c73" containerName="payload-processing" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095153 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b80f7431-03bd-4c63-8d31-ca3b70d68c73" containerName="payload-processing" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095162 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095168 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095174 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095178 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095186 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095191 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095197 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" Aug 04 01:05:17.095238 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095202 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095255 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095263 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095269 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b80f7431-03bd-4c63-8d31-ca3b70d68c73" containerName="payload-processing" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095276 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095282 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095287 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095292 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5184eb1-98c7-4bcf-b37e-3dc847c79a37" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095298 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ccbe229-aa81-42d1-915d-758ad4ca1321" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095304 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed002a2-9025-4a51-9cbb-6919bcb967a3" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095361 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095366 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095409 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b9dac14-724b-473c-91fd-c294923dd847" containerName="istio-proxy" Aug 04 01:05:17.095844 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.095417 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="63177a1f-0b99-4405-98c8-21a8574a74e6" containerName="istio-proxy" Aug 04 01:05:17.099448 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.099432 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cdcm/must-gather-cllcw" Aug 04 01:05:17.101764 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.101738 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cdcm\"/\"openshift-service-ca.crt\"" Aug 04 01:05:17.101907 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.101738 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9cdcm\"/\"default-dockercfg-wr28w\"" Aug 04 01:05:17.101907 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.101809 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cdcm\"/\"kube-root-ca.crt\"" Aug 04 01:05:17.109706 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.109684 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cdcm/must-gather-cllcw"] Aug 04 01:05:17.169641 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.169604 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/efab88b8-c683-4435-9789-4d427137b74b-must-gather-output\") pod \"must-gather-cllcw\" (UID: \"efab88b8-c683-4435-9789-4d427137b74b\") " pod="openshift-must-gather-9cdcm/must-gather-cllcw" Aug 04 01:05:17.169641 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.169646 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frhm\" (UniqueName: \"kubernetes.io/projected/efab88b8-c683-4435-9789-4d427137b74b-kube-api-access-8frhm\") pod \"must-gather-cllcw\" (UID: \"efab88b8-c683-4435-9789-4d427137b74b\") " pod="openshift-must-gather-9cdcm/must-gather-cllcw" Aug 04 01:05:17.270958 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.270922 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/efab88b8-c683-4435-9789-4d427137b74b-must-gather-output\") pod \"must-gather-cllcw\" (UID: \"efab88b8-c683-4435-9789-4d427137b74b\") " pod="openshift-must-gather-9cdcm/must-gather-cllcw" Aug 04 01:05:17.270958 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.270964 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frhm\" (UniqueName: \"kubernetes.io/projected/efab88b8-c683-4435-9789-4d427137b74b-kube-api-access-8frhm\") pod \"must-gather-cllcw\" (UID: \"efab88b8-c683-4435-9789-4d427137b74b\") " pod="openshift-must-gather-9cdcm/must-gather-cllcw" Aug 04 01:05:17.271279 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.271261 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/efab88b8-c683-4435-9789-4d427137b74b-must-gather-output\") pod \"must-gather-cllcw\" (UID: \"efab88b8-c683-4435-9789-4d427137b74b\") " pod="openshift-must-gather-9cdcm/must-gather-cllcw" Aug 04 01:05:17.278750 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.278720 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frhm\" (UniqueName: \"kubernetes.io/projected/efab88b8-c683-4435-9789-4d427137b74b-kube-api-access-8frhm\") pod \"must-gather-cllcw\" (UID: \"efab88b8-c683-4435-9789-4d427137b74b\") " pod="openshift-must-gather-9cdcm/must-gather-cllcw" Aug 04 01:05:17.408621 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.408588 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cdcm/must-gather-cllcw" Aug 04 01:05:17.537625 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.537557 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cdcm/must-gather-cllcw"] Aug 04 01:05:17.539612 ip-10-0-132-168 kubenswrapper[2552]: W0804 01:05:17.539584 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefab88b8_c683_4435_9789_4d427137b74b.slice/crio-eef21b306b57bd5d6c765396a2e4d7c9589c60d1656402e84dfe54cd7846a39b WatchSource:0}: Error finding container eef21b306b57bd5d6c765396a2e4d7c9589c60d1656402e84dfe54cd7846a39b: Status 404 returned error can't find the container with id eef21b306b57bd5d6c765396a2e4d7c9589c60d1656402e84dfe54cd7846a39b Aug 04 01:05:17.812079 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:17.811986 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cdcm/must-gather-cllcw" event={"ID":"efab88b8-c683-4435-9789-4d427137b74b","Type":"ContainerStarted","Data":"eef21b306b57bd5d6c765396a2e4d7c9589c60d1656402e84dfe54cd7846a39b"} Aug 04 01:05:18.822999 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:18.822966 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cdcm/must-gather-cllcw" event={"ID":"efab88b8-c683-4435-9789-4d427137b74b","Type":"ContainerStarted","Data":"9f8602aa935dbb6a05e55bfed2b1de7b258ce0336ca229797c6b8fab675f9f8e"} Aug 04 01:05:18.823486 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:18.823013 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cdcm/must-gather-cllcw" event={"ID":"efab88b8-c683-4435-9789-4d427137b74b","Type":"ContainerStarted","Data":"a85ea31b95bf2f3584917faab4daa1e1e754f5e4f5975703b2b0de6fc3c22bb3"} Aug 04 01:05:18.839073 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:18.839007 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9cdcm/must-gather-cllcw" podStartSLOduration=1.053580603 podStartE2EDuration="1.838985263s" totalImagesPullingTime="785.40466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:05:18.838412495 +0000 UTC m=+3757.501321974" watchObservedRunningTime="2026-08-04 01:05:18.838985263 +0000 UTC m=+3757.501894741" Aug 04 01:05:19.853842 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:19.853811 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nhd8z_c93f255c-fe84-43d8-8ebd-4c54a072e1f8/global-pull-secret-syncer/0.log" Aug 04 01:05:19.995892 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:19.995844 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8vz67_9a77bedb-69cf-4cd2-8a73-79829ed0c663/konnectivity-agent/0.log" Aug 04 01:05:20.069125 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:20.069095 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-168.ec2.internal_7c14b132a47c877cb6e709b2d262ede1/haproxy/0.log" Aug 04 01:05:25.518605 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:25.518501 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-v2sg7_e5d1cd67-c23b-4afc-829c-42b95129c58d/manager/0.log" Aug 04 01:05:25.541694 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:25.541502 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-fwkml_8c04aaf8-469f-4662-ab08-d7c7f59fa482/manager/0.log" Aug 04 01:05:25.740580 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:25.740532 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-bs9h7_b9e19202-0064-4b6a-9470-eeeb6094955e/manager/0.log" Aug 04 01:05:25.869529 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:25.869463 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-wdpgm_63cfe342-44f8-4c23-8ad7-b1aa22b1a943/manager/0.log" Aug 04 01:05:27.765841 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:27.765779 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-j7krx_90e765d3-ddca-4bf1-867e-b1468bd64b63/kube-state-metrics/0.log" Aug 04 01:05:27.834420 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:27.834360 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-j7krx_90e765d3-ddca-4bf1-867e-b1468bd64b63/kube-rbac-proxy-main/0.log" Aug 04 01:05:27.856056 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:27.856027 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-j7krx_90e765d3-ddca-4bf1-867e-b1468bd64b63/kube-rbac-proxy-self/0.log" Aug 04 01:05:27.913624 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:27.913590 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-2vl2c_febed18d-372c-4b67-8396-bff767963966/monitoring-plugin/0.log" Aug 04 01:05:28.012588 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:28.012560 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qvgp4_45955895-bd01-4566-8ace-f5ccdc90e24a/node-exporter/0.log" Aug 04 01:05:28.035100 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:28.035021 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qvgp4_45955895-bd01-4566-8ace-f5ccdc90e24a/kube-rbac-proxy/0.log" Aug 04 01:05:28.055081 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:28.055045 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qvgp4_45955895-bd01-4566-8ace-f5ccdc90e24a/init-textfile/0.log" Aug 04 01:05:29.991029 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:29.990981 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5"] Aug 04 01:05:29.996840 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:29.996808 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.003333 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.003305 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5"] Aug 04 01:05:30.112906 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.112849 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-proc\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.113093 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.113062 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-lib-modules\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.113222 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.113172 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-podres\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.113426 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.113407 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2wz\" (UniqueName: \"kubernetes.io/projected/8bb04902-eac6-449b-8221-00135875f194-kube-api-access-tb2wz\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.113603 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.113588 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-sys\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.214367 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.214314 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-podres\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.214556 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.214378 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2wz\" (UniqueName: \"kubernetes.io/projected/8bb04902-eac6-449b-8221-00135875f194-kube-api-access-tb2wz\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.214556 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.214422 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-sys\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.214556 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.214481 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-proc\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.214742 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.214558 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-lib-modules\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.214800 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.214742 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-lib-modules\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.214883 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.214835 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-podres\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.214950 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.214925 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-proc\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.215122 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.215096 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bb04902-eac6-449b-8221-00135875f194-sys\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.223897 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.223849 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2wz\" (UniqueName: \"kubernetes.io/projected/8bb04902-eac6-449b-8221-00135875f194-kube-api-access-tb2wz\") pod \"perf-node-gather-daemonset-888n5\" (UID: \"8bb04902-eac6-449b-8221-00135875f194\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.313089 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.313003 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.496713 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.495527 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5"] Aug 04 01:05:30.497326 ip-10-0-132-168 kubenswrapper[2552]: W0804 01:05:30.497281 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8bb04902_eac6_449b_8221_00135875f194.slice/crio-9d4e7ecfc4c72e319190ae9362b343b1c858c0c473f98bbaa7670e863fd80c91 WatchSource:0}: Error finding container 9d4e7ecfc4c72e319190ae9362b343b1c858c0c473f98bbaa7670e863fd80c91: Status 404 returned error can't find the container with id 9d4e7ecfc4c72e319190ae9362b343b1c858c0c473f98bbaa7670e863fd80c91 Aug 04 01:05:30.919144 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.919105 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" event={"ID":"8bb04902-eac6-449b-8221-00135875f194","Type":"ContainerStarted","Data":"9d19cf7a7034d60d1aec2e3933d3eb9b2d6762a7eca9caea9745319fe6410e3c"} Aug 04 01:05:30.919317 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.919176 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" event={"ID":"8bb04902-eac6-449b-8221-00135875f194","Type":"ContainerStarted","Data":"9d4e7ecfc4c72e319190ae9362b343b1c858c0c473f98bbaa7670e863fd80c91"} Aug 04 01:05:30.919317 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.919210 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:30.935875 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:30.935795 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" podStartSLOduration=1.935773555 podStartE2EDuration="1.935773555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 01:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:05:30.93359151 +0000 UTC m=+3769.596501005" watchObservedRunningTime="2026-08-04 01:05:30.935773555 +0000 UTC m=+3769.598683034" Aug 04 01:05:32.574775 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:32.574747 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bn6k_29ed5491-dd76-40c3-b091-cd27679ab793/dns/0.log" Aug 04 01:05:32.595752 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:32.595722 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bn6k_29ed5491-dd76-40c3-b091-cd27679ab793/kube-rbac-proxy/0.log" Aug 04 01:05:32.751664 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:32.751630 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v8btt_6920ae2c-7e06-47e6-9101-47f93c1b3226/dns-node-resolver/0.log" Aug 04 01:05:33.295315 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:33.295209 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-75ff4b669d-kbgg5_1ac3d4fe-04d7-46bf-96f8-81dec40df02f/registry/0.log" Aug 04 01:05:33.354766 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:33.354731 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fkmvm_cc72d8b6-1df2-48fb-b76e-671a7c4b905e/node-ca/0.log" Aug 04 01:05:34.299735 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:34.299698 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q_26863ba2-e38c-48b9-ae4e-95ec614716ff/istio-proxy/0.log" Aug 04 01:05:34.649555 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:34.649517 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-554b4579cb-f2ksc_f20a59f4-5d4e-4796-b7d8-60b9587e8b3d/kube-auth-proxy/0.log" Aug 04 01:05:34.709133 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:34.709099 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8cwk8_9c4e0926-157d-4f36-874d-bf2af64dbc79/istio-proxy/0.log" Aug 04 01:05:34.717283 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:34.717262 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8cwk8_9c4e0926-157d-4f36-874d-bf2af64dbc79/istio-proxy/1.log" Aug 04 01:05:34.790693 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:34.790656 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-j2q5h_d1436586-9a7f-4366-a7d1-c504cbac172e/payload-processing/0.log" Aug 04 01:05:35.637189 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:35.637164 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wsntm_1ed4fcbd-979e-40a5-be0d-11d65f7c4053/serve-healthcheck-canary/0.log" Aug 04 01:05:36.192430 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:36.192397 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kdbcj_2545bce7-2c88-4073-9cd9-733ad69b540c/kube-rbac-proxy/0.log" Aug 04 01:05:36.211675 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:36.211651 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kdbcj_2545bce7-2c88-4073-9cd9-733ad69b540c/exporter/0.log" Aug 04 01:05:36.231663 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:36.231634 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kdbcj_2545bce7-2c88-4073-9cd9-733ad69b540c/extractor/0.log" Aug 04 01:05:36.937325 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:36.937295 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-888n5" Aug 04 01:05:38.323744 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:38.323715 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-h7qtw_8e648d96-5199-4788-ab69-79b1e3cad502/manager/0.log" Aug 04 01:05:38.666515 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:38.666482 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lnpqr_fe9f0525-cf27-4a1d-9722-698306049b57/manager/0.log" Aug 04 01:05:39.194138 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:39.194056 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ddshn_bbce56ae-aa73-41f1-9e9e-08a3573fc8bd/server/0.log" Aug 04 01:05:39.309710 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:39.309679 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gxlrp_08461dce-ee65-4059-9985-ca2af7252bcb/manager/0.log" Aug 04 01:05:39.559809 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:39.559677 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-wgcxn_9efde704-d281-4e4a-8164-160af9c7033a/manager/0.log" Aug 04 01:05:39.583779 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:39.583580 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-wgcxn_9efde704-d281-4e4a-8164-160af9c7033a/copy-manifests/0.log" Aug 04 01:05:40.224476 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:40.224387 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q_26863ba2-e38c-48b9-ae4e-95ec614716ff/istio-proxy/0.log" Aug 04 01:05:40.568707 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:40.568638 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-554b4579cb-f2ksc_f20a59f4-5d4e-4796-b7d8-60b9587e8b3d/kube-auth-proxy/0.log" Aug 04 01:05:40.600504 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:40.600474 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8cwk8_9c4e0926-157d-4f36-874d-bf2af64dbc79/istio-proxy/0.log" Aug 04 01:05:40.608127 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:40.608107 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8cwk8_9c4e0926-157d-4f36-874d-bf2af64dbc79/istio-proxy/1.log" Aug 04 01:05:40.660239 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:40.660203 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-j2q5h_d1436586-9a7f-4366-a7d1-c504cbac172e/payload-processing/0.log" Aug 04 01:05:40.918028 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:40.917995 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tp5f4_1132b218-176c-49ee-9694-eb122741113a/openshift-lws-operator/0.log" Aug 04 01:05:42.121915 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:42.121883 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-fpk5q_26863ba2-e38c-48b9-ae4e-95ec614716ff/istio-proxy/0.log" Aug 04 01:05:42.474017 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:42.473992 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-554b4579cb-f2ksc_f20a59f4-5d4e-4796-b7d8-60b9587e8b3d/kube-auth-proxy/0.log" Aug 04 01:05:42.508550 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:42.508524 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8cwk8_9c4e0926-157d-4f36-874d-bf2af64dbc79/istio-proxy/0.log" Aug 04 01:05:42.516633 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:42.516607 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8cwk8_9c4e0926-157d-4f36-874d-bf2af64dbc79/istio-proxy/1.log" Aug 04 01:05:42.575824 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:42.575794 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-j2q5h_d1436586-9a7f-4366-a7d1-c504cbac172e/payload-processing/0.log" Aug 04 01:05:47.002740 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:47.002669 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4wxgh_f6725222-fca9-49d7-b2c5-9b172b300071/kube-multus/0.log" Aug 04 01:05:47.232379 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:47.232348 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqmnm_a966195c-aac5-44da-9599-8bad5ef0e2c3/kube-multus-additional-cni-plugins/0.log" Aug 04 01:05:47.252424 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:47.252395 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqmnm_a966195c-aac5-44da-9599-8bad5ef0e2c3/egress-router-binary-copy/0.log" Aug 04 01:05:47.271553 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:47.271495 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqmnm_a966195c-aac5-44da-9599-8bad5ef0e2c3/cni-plugins/0.log" Aug 04 01:05:47.292113 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:47.292095 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqmnm_a966195c-aac5-44da-9599-8bad5ef0e2c3/bond-cni-plugin/0.log" Aug 04 01:05:47.312402 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:47.312373 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqmnm_a966195c-aac5-44da-9599-8bad5ef0e2c3/routeoverride-cni/0.log" Aug 04 01:05:47.331961 ip-10-0-132-168 kubenswrapper[2552]: I0804 01:05:47.331942 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqmnm_a966195c-aac5-44da-9599-8bad5ef0e2c3/whereabouts-cni-bincopy/0.log"